Methods for augmenting a surgical field with virtual guidance and tracking and adapting to deviation from a surgical plan

ABSTRACT

One variation of a method includes: accessing a virtual patient model defining a target resected contour of a hard tissue of interest; after resection of the hard tissue of interest during a surgical operation, accessing an optical scan recorded by an optical sensor facing a surgical field occupied by a patient, detecting a set of features representing the patient in the optical scan, registering the virtual patient model to the hard tissue of interest in the surgical field based on the set of features, and detecting an actual resected contour of the hard tissue of interest in the optical scan; and calculating a spatial difference between the actual resected contour of the hard tissue of interest and the target resected contour of the hard tissue of interest represented in the virtual patient model registered to the hard tissue of interest in the surgical field.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is a continuation-in-part application of U.S. patentapplication Ser. No. 15/594,623, filed on 14 May 2017, which claims thebenefit of U.S. Provisional Application No. 62/363,022, filed on 15 Jul.2016, and which is a continuation-in-part application of U.S. patentapplication Ser. No. 15/499,046, filed on 27 Apr. 2017, which claims thebenefit of U.S. Provisional Application No. 62/328,330, filed on 27 Apr.2016, and U.S. Provisional Application No. 62/363,022, filed on 15 Jul.2016, all of which are incorporated in their entireties by thisreference.

Furthermore, this Application claims the benefit of U.S. ProvisionalApplication No. 62/612,895, filed on 2 Jan. 2018, and U.S. ProvisionalApplication No. 62/612,901, filed on 2 Jan. 2018, both of which areincorporated in their entireties by this reference.

TECHNICAL FIELD

This invention relates generally to the field of augmented reality andmore specifically to a new and useful method for registering features ofa patient's body within a surgical field to provide virtual guidance inthe field of augmented reality.

BRIEF DESCRIPTION OF THE FIGURES

FIGS. 1A and 1B are flowchart representations of a method;

FIG. 2 is a flowchart representation of one variation of the method;

FIG. 3 is a flowchart representation of one variation of the method;

FIG. 4 is a flowchart representation of one variation of the method; and

FIG. 5 is a flowchart representation of one variation of the method.

DESCRIPTION OF THE EMBODIMENTS

The following description of embodiments of the invention is notintended to limit the invention to these embodiments but rather toenable a person skilled in the art to make and use this invention.Variations, configurations, implementations, example implementations,and examples described herein are optional and are not exclusive to thevariations, configurations, implementations, example implementations,and examples they describe. The invention described herein can includeany and all permutations of these variations, configurations,implementations, example implementations, and examples.

1. Method

As shown in FIGS. 1A and 1B, a method S100 for registering features of apatient in a surgical field includes accessing a virtual patient modelrepresenting a hard tissue of interest of the patient in Block S120, thevirtual patient model generated from a pre-operative scan of the hardtissue of interest of the patient. The method S100 also includes, duringa first period of time succeeding incision of the patient proximal thehard tissue of interest and prior to resection of the hard tissue ofinterest within a surgical operation: accessing a first sequence ofoptical scans recorded by an optical sensor facing a surgical fieldoccupied by the patient in Block S130; detecting a first contour of thehard tissue of interest in the first sequence of optical scans in BlockS132; registering virtual hard tissue features defined in the virtualpatient model to the first contour of the hard tissue of interest inBlock S134; and detecting a set of intermediate features, on the patientand proximal the hard tissue of interest, in the first sequence ofoptical scans in Block S136. The method S100 further includes deriving aspatial relationship between the set of intermediate features and thevirtual patient model based on registration of the virtual patient modelto the hard tissue of interest in Block S140. The method S100 alsoincludes, during a second period of time succeeding resection of thehard tissue of interest within the surgical operation: accessing asecond sequence of optical scans recorded by the optical sensor in BlockS150; detecting the set of intermediate features in the second sequenceof optical scans in Block S156; registering the virtual patient model tothe hard tissue of interest based on the spatial relationship and theset of intermediate features detected in the second sequence of opticalscans in Block S154; and detecting a second contour of the hard tissueof interest in the second sequence of optical scans in Block S152. Themethod S100 further includes detecting a spatial difference betweenvirtual hard tissue features defined in the virtual patient model andthe second contour of the hard tissue of interest detected in the secondsequence of optical scans in Block S160.

One variation of the method S100 includes accessing a virtual anatomicalmodel representing a hard tissue of interest in human anatomy in BlockS120. This variation of the method S100 also includes, during a firstperiod of time succeeding incision of the patient proximal the hardtissue of interest and prior to resection of the hard tissue of interestwithin a surgical operation: accessing a first sequence of optical scansrecorded by an optical sensor facing the surgical field occupied by thepatient in Block S130; detecting a first contour of the hard tissue ofinterest in the first sequence of optical scans in Block S132;registering virtual hard tissue features defined in the virtualanatomical model to the first contour of the hard tissue of interest inBlock S134; and detecting a set of intermediate features, on the patientand proximal the hard tissue of interest, in the first sequence ofoptical scans in Block S136. This variation of the method S100 furtherincludes deriving a spatial relationship between the set of intermediatefeatures and the virtual anatomical model based on registration of thevirtual anatomical model to the hard tissue of interest in Block S140.This variation of the method S100 also includes, during a second periodof time succeeding resection of the hard tissue of interest within thesurgical operation: accessing a second sequence of optical scansrecorded by the optical sensor in Block S150; detecting a second contourof the hard tissue of interest in the second sequence of optical scansin Block S152; detecting the set of intermediate features in the secondsequence of optical scans in Block S156; and, in response to presence ofthe second contour in place of the first contour in the second sequenceof optical scans, registering the virtual anatomical model to the hardtissue of interest based on the spatial relationship and the set ofintermediate features detected in the second sequence of optical scansin Block S154. Finally, this variation of the method S100 also includesdetecting a spatial difference between virtual hard tissue featuresdefined in the virtual anatomical model and the second contour of thehard tissue of interest detected in the second sequence of optical scansin Block S160.

Another variation of the method S100 shown in FIG. 3 includes accessinga virtual patient model defining a target resected contour of a hardtissue of interest in Block S120. This variation of the method S100 alsoincludes, during a first period of time succeeding resection of the hardtissue of interest within a surgical operation: accessing a firstsequence of optical scans recorded by an optical sensor facing asurgical field occupied by a patient in Block S150; detecting a set offeatures representing the patient in the first sequence of optical scansin Block S156; registering the virtual patient model to the hard tissueof interest in the surgical field based on the set of features in BlockS154; and detecting an actual resected contour of the hard tissue ofinterest in the first sequence of optical scans in Block S152. Thisvariation of the method S100 further includes: calculating a spatialdifference between the actual resected contour of the hard tissue ofinterest detected in the first sequence of optical scans and the targetresected contour of the hard tissue of interest represented in thevirtual patient model registered to the hard tissue of interest in thesurgical field in Block S160; and presenting the spatial difference to asurgeon during the surgical operation in Block S170.

Yet another variation of the method S100 includes accessing a virtualpatient model defining a target position of a artificial implant on ahard tissue of interest in Block S120. This variation of the method S100also includes, during a first period of time succeeding placement of theartificial implant on the hard tissue of interest within a surgicaloperation: accessing a first sequence of optical scans recorded by anoptical sensor facing a surgical field occupied by a patient in BlockS150; detecting a set of features representing the patient in the firstsequence of optical scans in Block S156; registering the virtual patientmodel to the hard tissue of interest in the surgical field based on theset of features in Block S154; and detecting an actual position of theartificial implant on the hard tissue of interest in the first sequenceof optical scans in Block S152. This variation of the method S100further includes: calculating a spatial difference between the actualposition of the artificial implant on the hard tissue of interestdetected in the first sequence of optical scans and the target positionof the artificial implant on the hard tissue of interest represented inthe virtual patient model registered to the hard tissue of interest inthe surgical field in Block S160; and presenting the spatial differenceto a surgeon during the surgical operation in Block S170.

2. Applications: Registration

As shown in FIGS. 1A and 1B, a computer system can execute Blocks of themethod S100 to access and transform scan data of a hard tissue ofinterest (e.g., bone) of a patient into a virtual patient modelrepresenting the hard tissue of interest prior to a surgical operationon the patient. For example, the computer system can generate a virtualpatient model depicting the patient's left femur and left tibia prior toa left knee replacement. Later, during the surgical operation, thecomputer system can access optical scan data from an optical sensor(e.g., a LIDAR or other depth sensor, color camera, stereoscopic camera,thermographic camera, multispectral camera) arranged in the surgicalfield and sequentially narrow objects detected in these optical scandata down to the patient's hard tissue of interest, including: firstidentifying the patient generally (e.g., by detecting the patient'shead, feet, and front or back side); identifying a region of the patientpredicted to contain the hard tissue of interest; and coarselyregistering the virtual patient model to this region of the patient. Asthe surgeon incises the patient near the hard tissue of interest, thecomputer system can verify that red pixels depicting blood and/or muscletissue in the next optical scan align with the region of the patientpredicted to contain the hard tissue of interest. As the surgeondisplaces soft tissue to reveal the hard tissue of interest, thecomputer system can: detect light-colored (e.g., approximately white)pixels depicting bone in the next optical scan; extractthree-dimensional (“3D”) anatomical features representing this bonesurface, which represented unique hard tissue anatomy of the patient;and align (or “snap”) the virtual representation of the correspondingbone in the virtual patient model to these 3D anatomical features,thereby aligning the virtual patient model to the hard tissue ofinterest detected in the surgical field.

Furthermore, if the computer system detects a difference between thevirtual patient model and the 3D features of the hard tissue of interestdetected in the surgical field, the computer system can also modify thevirtual patient model to better resemble these 3D features of the hardtissue of interest. The computer system can therefore detect and handlethese 3D features of the hard tissue of interest as an initial “groundtruth” of the patient.

However, because this hard tissue of interest may change as the surgeonresects portions of the hard tissue of interest and/or installsartificial components on or near the hard tissue of interest, these 3Dfeatures of the patient's hard tissue of interest may be removed orobscured from the optical sensor. Therefore, once the computer systemhas aligned the virtual patient model to the hard tissue of interest,the computer system can also define a constellation of intermediatefeatures—remote from the hard tissue of interest—that bridgeregistration of the virtual patient model and the hard tissue ofinterest.

For example, for a left knee replacement, the computer system can:define a constellation of intermediate features for the patient's leftfemur that includes a mechanical axis of the patient's left femur, aglobal 3D skin surface profile of the patient's left upper leg, and aset of freckles, moles, or other superficial skin features on thepatient's left upper leg; and define a constellation of intermediatefeatures for the patient's left tibia that includes a mechanical axis ofthe patient's left tibia, a global 3D skin surface profile of thepatient's left lower leg, and a set of freckles, moles, or othersuperficial skin features on the patient's left lower leg. Thus, as thepatient's left femoral condyles and left tibial plateau are resectedduring the knee replacement surgery, the computer system can: continueto access optical scan data recorded by the optical sensor; track thepatient, the patient's left leg, and bone surfaces in the patient's leftknee in the surgical field based on features extracted from this opticalscan data; align the virtual patient model of the patient's left femurand left tibia to corresponding bone features detected in the surgicalfield while these bone features are present and not obscured; andtransition to aligning the virtual patient model of the patient's leftfemur and left tibia to corresponding constellations of intermediatefeatures detected in the surgical field once the corresponding bonefeatures are resected or are otherwise obscured from the optical sensor.

Therefore, once the patient's hard tissue of interest has been modified(e.g., resected or modified via installation of an artificialcomponent), the computer system can transition to: handling the virtualpatient model as “ground truth” for the patient; and registering thevirtual patient model to the patient based on the constellation ofintermediate features. In particular, once the surgeon resects the hardtissue of interest, the computer system can implement the virtualpatient model as the virtual “ground truth” representation of thepatient's anatomy—registered to other hard and/or soft tissuefeatures—for all subsequent steps of the surgery such that this groundtruth representation of the patient defines a preoperative anatomicalstate the patient regardless of changes made to the patient's anatomyduring the surgery, thereby enabling the surgeon: to “look back” toquantify actual changes in the patient's anatomy during the surgery; andto “look forward” to planned future changes to the patient's anatomyduring the surgery based on this virtual ground truth representation ofthe patient's anatomy.

The computer system can also: generate augmented reality framesrepresenting the virtual patient model aligned with the patient'sanatomy; serve these augmented reality frames to a display (e.g., aheads-up or eyes-up augmented reality display worn by a surgeon duringthe operation) in real-time in order to preserve a visual representationof the pre-operative state of the hard tissue of interest—as representedin the virtual patient model—for the surgeon as the surgeon modifies thehard tissue of interest throughout the surgical operation. The surgeonmay therefore reference these augmented reality frames—overlaid on thepatient's hard tissue of interest—to quickly visualize real changes tothe hard tissue of interest from its pre-operative state.

Therefore, the computer system can execute Blocks of the method S100throughout a real surgical operation in order to preserve an accuraterepresentation of the original, unmodified hard tissue ofinterest—aligned to corresponding real features on the patient's body,even as some of these real features change. The computer system can thencharacterize differences between this virtual patient model and thepatient's hard tissue of interest—detected in later scan data recordedby the optical sensor—as the hard tissue of interest is modifiedthroughout the surgical operation and thus return quantitative guidanceto the surgeon regarding position, orientation, and magnitude, etc. ofabsolute changes to the hard tissue of interest. The computer system canalso: detect differences between these absolute changes to (e.g.,resection of) the hard tissue of interest and target changes to the hardtissue of interest defined in a surgical plan registered to the virtualpatient model; and return quantitative metrics regarding differencesbetween these actual and target changes, thereby enabling the surgeon toconfirm intent of such differences or further modify the hard tissue ofinterest to achieve better alignment with the surgical plan.Additionally or alternatively, the computer system can: detectdifferences between the absolute position of a surgical implantinstalled on the hard tissue of interest and a target position of thesurgical implant defined in the surgical plan registered to the virtualpatient model; and return quantitative metrics regarding differencesbetween these actual and target surgical outcomes, thereby enabling thesurgeon to confirm intent of such differences or modify the position ofthe surgical implant to achieve better alignment with the surgical plan.

Blocks of the method S100 are described herein in the context of a kneereplacement surgery. However, Blocks of the method S100 can be executedby a computer system to register a virtual patient model to a patient'shard and/or soft tissue features and to preserve this virtual patientmodel—registered to the patient's real tissue—as a virtual ground truthstate of the patient's original hard tissue of interest in any othersurgical or medical application, such as: a hip replacement operation; arotator cuff repair surgery; a heart valve replacement operation; acarpal tunnel release surgery; a cataract removal procedure; or asurgical repair of a comminuted or open fracture; etc. Furthermore,Blocks of the method S100 are described herein in the context ofregistering a virtual model of a hard tissue of interest to hard andsoft tissue features detected in the surgical field. However, similarmethods and techniques can be executed by the computer system toregister a soft tissue of interest (e.g., an aortic valve, an artery, apancreas) to other hard and/or soft features within a patient's body.

The method is also described below as executed by the computer system togenerate augmented reality frames for presentation to a local surgeon inreal-time during the surgery—such as through an augmented realityheadset worn by the local surgeon or other display located in theoperating room—to provide real-time look-back and look-forward guidanceto the local surgeon. However, the computer system can implement similarmethods and techniques to generate virtual reality frames depicting bothreal patient tissue and virtual content (e.g., target resected contoursof tissues of interest defined in a virtual patient model thusregistered to the real patient tissue) and to serve these virtualreality frames to a remote surgeon (or remote student). For example, thecomputer system can generate and serve such virtual reality frames to avirtual reality headset worn by a remote surgeon in real-time during thesurgery in order to enable the remote surgeon to: monitor the surgery;manually adjust parameters of the surgery or surgical plan; selectivelyauthorize next steps of the surgical plan; and/or serve real-timeguidance to the local surgeon.

3. Applications: Deviations from Surgical Plan

Furthermore, the computer system can execute Blocks of the method S100to track compliance with and/or deviations from a surgical planprescribed for the patient by a surgeon, such as prior to a surgery orin real-time during the surgery, as shown in FIGS. 2 and 3. Inparticular, by preserving registration of the virtual patient model—suchas including virtual representations of an unresected hard tissue ofinterest of the patient, a target resected contour of the patient,and/or a target position of a surgical implant—to the patient's hardtissue of interest and tracking the hard tissue of interest throughoutthe surgery, the computer system can detect differences between actualand target resected contours of the hard tissue of interest anddifferences between actual and target positions of a surgical implant onor near the hard tissue of interest during the surgery. The computersystem can return these differences to the surgeon in real-time—such asthrough augmented reality frames rendered on an augmented realityheadset worn by the surgeon—in order to guide the surgeon in correctingthe actual resected contour of the hard tissue of interest or adjustinga position of the surgical implant on the hard tissue of interest beforemoving to a next step of the surgical operation. The computer system canalso adapt subsequent steps of the surgical plan to account for priordeviations from the surgical plan, such as to minimize cumulativedeviation that may negatively affect the patient's surgical outcome. Forexample, the computer system can generate a sequence of augmentedreality (“AR”) frames aligned to the hard tissue of interest in thesurgeon's field of view and serve these augmented reality frames to anAR headset or AR glasses (or to another display in the surgical field)in order to visually indicate to the surgeon compliance with and/ordeviation from steps of the surgical plan.

In particular, the computer system can access a surgical plan—such asdefined by the computer or entered manually by a surgeon, radiologist,engineer, technician, etc. before or during the surgery—defining asequence of target resected contours (or “resected contours”) of apatient's hard tissue of interest resulting from a sequence of surgicalsteps performed on the hard tissue of interest during an upcomingsurgery. During the subsequent surgery, the computer system can: accessoptical scan data from an optical sensor arranged near the surgicalfield; implement computer vision techniques to detect a hard tissue ofinterest (or other tissues surrounding the hard tissue of interest) inthe surgical field; and virtually align a virtual representation of theunresected hard tissue of interest with the hard tissue of interestdetected in the surgical field. Throughout the surgery, the computersystem can: continue to capture and/or access optical scan data of thesurgical field via the optical sensor (e.g., at a rate of 24frames-per-second); and extract actual contours of the hard tissue ofinterest in these optical scan data as the surgeon incises soft tissuenear the hard tissue of interest, resects the hard tissue of interest,and eventually locates a surgical implant on the hard tissue ofinterest. In response to differences between the actual resected contourof the hard tissue of interest detected in these optical scan data andthe target resected contour—represented in the *virtual patient model*and/or defined by the surgical plan—the computer system can either:prompt the surgeon to refine the actual resected contour to achievegrater alignment with the target resected contour if the actual resectedcontour extends beyond the target resected contour; or update subsequentsteps of the surgical plan to compensate for excessive removal ofmaterial from the hard tissue of interest if the target resected contourextends beyond the actual resected contour. Alternatively, if thesurgeon confirms the actual resected contour of the hard tissue ofinterest, the computer system can update subsequent steps of thesurgical plan to compensate for this deviation from the originalsurgical plan. Therefore, computer system can execute Blocks of themethod S100 to detect intended and unintended deviations from anoriginal surgical plan and then modify the original surgical plan tocompensate for these deviations and thus limit cumulative deviation fromthe original surgical plan upon completion of the surgery.

For example, the computer system can determine—based on a differencebetween a virtual patient model and a hard tissue of interest detectedin optical scan data of a surgical field—that a surgeon (unintentionallyor unknowingly) resected a tibial plateau two degrees offset from aplanned cut to the tibial plateau as defined in a surgical plan. Thecomputer system can then prompt or guide the surgeon to recut the tibialplateau in order to reduce this offset. Alternatively, if the surgeonconfirms the offset from the surgical plan, the computer system caninstead modify the surgical plan automatically to adjust a targetcontour of the adjacent femoral head of the patient by two degrees inthe opposite direction in order to compensate for deviation from thesurgical plan at the tibial plateau. Yet alternatively, the computersystem can modify the surgical plan to offset the trajectory of a boreinto the adjacent femur—to accept an artificial femoral component—by twodegrees from normal to the actual resected contour to the tibial plateausuch that the artificial femoral component properly mates with anartificial tibial component installed on the offset resected contour ofthe tibial plateau. In particular, the computer system can adapt thesurgical plan to counteract this deviation at the tibial plateau. Yetalternatively, the computer system can: determine that this differencebetween the actual and target resected contours of the tibial plateauprescribed in the surgical plan falls within an acceptable tolerancerange defined for this step of the surgery; record this deviation withina log file for the surgical operation; and repeat this process for othersteps of the surgery.

The computer system can also automatically modify a surgical plan tocorrect or accommodate for intentional deviations from the surgical planperformed by the surgeon, thereby empowering the surgeon to adapt thesurgical plan inter-operatively. For example, the computer system canreceive a command from the surgeon to rotate a target resected contourto the tibial plateau of the patient—as defined in the original surgicalplan—by one degree and to move the target resected contour fivemillimeters distally, such as after the surgeon has opened the patient'sknee and inspected the patient's tibia and femur bone structures. Thecomputer system can then modify the surgical plan accordingly, such asby modifying the target resected contour of the patient's femoralcondyle defined in the surgical plan to preserve parallelism to andcoaxiality with the tibial plateau. Therefore, the computer system canenable a surgeon to manually adjust a current stop of the surgical planinter-operatively and then automatically adapt remaining steps of thesurgical plan to achieve an acceptable patient outcome accordingly.

Based on historical deviations from a particular surgical plan duringone surgery and/or across multiple surgeries by a particular surgeon,the computer system can predict deviations in future surgeries andpreemptively adapt surgical plans for those future surgeries tocompensate for these predicted future deviations. For example, based onhistorical surgical data, the computer system can determine that aparticular surgeon typically cuts the tibial plateau within a toleranceof five degrees of a target resected contour as defined in the surgeon'ssurgical plans for a knee replacement surgery. The computer system canalso determine that most actual resected contours to tibial plateausfall between three degrees and five degrees offset from the targetresected contour for this surgeon. The computer system can then predictthat future cuts to tibial plateaus performed by this surgeon are likelyto fall within three degrees and five degrees from the target resectedcontour defined in the surgeon's future surgical plans. The computersystem can then preemptively calculate a tolerance stackup for thesurgeon's knee replacement surgeries resulting from, for example,consistent five degree deviations in tibial plateau incisions and thenadapt surgical plans for future knee replacement surgeries to allow adeviation tolerance band of five degrees for tibial plateau incisionsbased on the tolerance stackup. Alternatively, the computer system cangenerate additional virtual guides or cut planes for the surgeon toimprove the surgeon's cut tolerance and similarly present augmentedreality frames depicting these virtual guides or cut planes to thesurgeon.

In another example, the computer system can extract data indicating thatseveral surgeons typically bore into the femur three degrees offset froma prescribed femoral bore incision in a particular surgical plan. Thecomputer system can adjust a particular surgical plan defined by one ofthese surgeons to offset the femoral bore incision by three degreesopposite typical off-axis boring performed by the several surgeons.Therefore, the computer system can preemptively adjust surgical plansaccording to historical surgical data to preempt deviations, accommodateor preemptively adapt to frequent deviations, and/or improve surgicalplans to reflect a consensus of surgeon preferences.

Furthermore, based on historical deviations from a particular surgicalplan during one surgery, across multiple surgeries of the same type by aparticular surgeon, or across a population of patients undergoing aparticular surgery type, the computer system can isolate the surgicalplan and/or surgical plan deviations predicted to yield positive (andnegative) outcomes for patients and guide surgeons in defining futuresurgical plans accordingly.

4. System

Blocks of the method S100 can be executed locally in an operating roomand/or remotely, such as: by a local computing device within anoperating room or within a hospital; by a remote computing device (e.g.,a remote server); and/or by a distributed computer network. Blocks ofthe method S100 can also be executed locally and/or remotely by acluster of computers. Blocks of the method S100 can additionally oralternatively be executed by an augmented reality headset, augmentedreality glasses, or other augmented reality device, such as worn by asurgeon in the operating room. A computing device executing Blocks ofthe method S100 can also interface with: an augmented reality device;one or more 2D color cameras, 3D cameras, and/or depth sensors (e.g., aLIDAR sensors, a structured light sensor); sensor-enabled surgicaltools; and/or other sensors and actuators within the operating room.

However, any other local, remote, or distributed computersystem—hereinafter referred to as “the computer system”—can executeBlocks of the method S100 substantially in real-time.

5. Virtual Patient Model

One variation of the method S100 shown in FIGS. 3 and 4 includes BlockS110, which recites, prior to the surgical operation: accessing apre-operative scan of the hard tissue of interest of the patient;extracting a virtual representation of the unresected contour of thehard tissue of interest from the pre-operative scan; generating avirtual representation of the target resected contour of the hard tissueof interest based on the virtual unresected contour of the hard tissueof interest and a pre-operative surgical plan defined by the surgeon;compiling the virtual representation of the unresected contour of thehard tissue of interest and the virtual representation of the targetresected contour of the hard tissue of interest into the virtual patientmodel; and storing the virtual patient model, in association with thepatient, in a database. Generally, in Block S110, the computer systemcan: access two-dimensional (“2D”) or three-dimensional (“3D”) MRI, CAT,X-ray (radiograph), or other scan data of all or a section of apatient's body designated for an upcoming surgery; and generate avirtual patient model of the patient based on these the scan data.

In one implementation, the computer system transforms pre-operative scandata (e.g., MRI scans, orthogonal X-rays images, and/or CT scans) of ahard tissue of interest into a virtual patient model representing thehard tissue of interest. For example, the computer system can access anMRI scan of a patient's left leg, including dimensionally-accuratedetails of bones (e.g., a femur and a tibia), tendons (e.g., a patellartendon), ligaments (e.g., an anterior cruciate ligament), muscles (e.g.,a quadriceps), other soft tissue (e.g., arteries, veins), and anenvelope (e.g., a 2D silhouette or 3D skin surface profile) of the leftleg. From the MRI scan, the computer system can generate a virtual scalerepresentation of the patient's left leg, such as in the form of avirtual patient model that includes a dimensionally-accurate contour,surface, and/or volumetric anatomical hard tissue and soft tissuefeatures of the patient's left leg.

In a similar implementation, the computer system can transform scan datainto a virtual patient model of the patient's body according to anabsolute scale for each bone, ligament, muscle, and/or other featuresrepresented within the scan data. Thus, the computer system can extractfrom the virtual patient model major dimensions, minor dimensions,contours, curvatures, etc. of anatomical components represented withinthe virtual patient model. For example, the computer system can combineorthogonal X-ray radiographs of a patient with a generic (parameterized)anatomical virtual patient model of a human anatomy. In order to yield acustom (patient-specific) virtual anatomical model reflective of thepatient's anatomy, the computer system can extract a first point fromthe set of orthogonal radiographs corresponding to a first discretelocation of the hard tissue of interest and query the generic virtualanatomical model for a first virtual point in the generic virtualanatomical model corresponding to the first point from the set oforthogonal radiographs. The first virtual point can be located in thegeneric virtual anatomical model by pattern matching the orthogonalradiographs with the generic virtual anatomical model to find similargeometry patterns (and shapes). In this example, the first point can bealigned adjacent a tibial plateau of the patient's tibia. The computersystem can identify a shape of the tibial plateau in the orthogonalradiographs by matching a similar shape of a tibial plateau in thegeneric anatomical model. The computer system can then locate the firstvirtual point relative to geometric features of the tibia in the genericvirtual patient model by identifying proximity of the first point togeometric features of the tibia in the orthogonal radiographs. Thecomputer system can further extract a second point from the set oforthogonal radiographs corresponding to a discrete location of the hardtissue of interest; and define a second virtual point in the genericvirtual anatomical model corresponding to the second point from the setof orthogonal radiographs. Based on a distance between the first andsecond points in the orthogonal radiographs, the computer system canscale the generic virtual anatomical model to define the custom virtualanatomical model by scaling a virtual distance between the first virtualpoint and the second virtual point in the custom virtual anatomicalmodel to correspond to the real distance between the first point and thesecond point in the set of orthogonal scans. Thus, a virtual distancebetween the first virtual point and the second virtual point can beproportional to the real distance in the set of orthogonal scans.

In another implementation, the computer system can implement templatematching techniques to match template tissue point clouds—labeled withone or more anatomical tissue labels—to tissue masses identified in the3D point cloud and transfer anatomical tissue labels from matchedtemplate tissue point clouds to corresponding tissue masses in the 3Dpoint cloud. Yet alternatively, the computer system can: implementcomputer vision techniques, such as edge detection or objectrecognition, to automatically detect distinct tissue masses in the scandata; present these distinct tissue masses in the scan data to thesurgeon through the physician portal; and write an anatomical tissuelabel to each distinct tissue mass in the 3D point cloud based onanatomical tissue labels manually entered or selected by the surgeonthrough the physician portal. However, the computer system can implementany other method or technique to label tissues within patient scan dataautomatically or with guidance from a surgeon.

In one variation, a reference marker of known dimension is placed in thefield of view of the scanner when the MRI, CAT, X-ray, or other scandata of the region of the patient's body is recorded. For example, three1″-diameter steel spheres can be placed at different (X, Y, Z) positionsaround a patient's left knee when the patient's left knee is imaged inan MRI scanner. When analyzing an MRI scan to generate a surgical plan,the computer system can interpolate real dimensions of the patient'stissues (e.g., general and feature-specific length, width, depth of thetibia, femur, patella, tibial condyle, and femoral condyle, etc.) basedon known dimensions of the reference marker(s). The computer system canlabel regions of patient tissues with these dimensions and/or can scaleor modify the virtual patient model into alignment with these dimensionsextracted from the patient scan data.

In another variation, by assembling data from a plurality of scanscapturing anatomical components (i.e., a joint) of the patient's body invarious positions, the computer system can extract a range of motion andrelative angles between anatomical components represented in the scans.Then, the computer system can define ranges of motion and relativeangles between virtual anatomical components represented in the virtualpatient model accordingly. From the virtual patient model, the computersystem can define constraint parameters and extract reasonable (orplausible) positions of the anatomical components in real space and,therefore, facilitate registration of the anatomical components asdescribed below. For example, the computer system can access scan dataof a knee (and areas surrounding the knee) bent to 30°, 45°, 90°, and120°. Based on the scans, the computer system can extract data such asvarus and/or valgus articulation of the tibia relative to the femur;degree of hyperextension of the tibia relative to the femur; and/orrange of motion of the knee (e.g., between thirty to ninety degrees).The computer system can then input this data as a parameter for thevirtual patient model, such that the virtual patient model reflectsanatomical dimension, articulation, contours, range of motion, etc.

However, the computer system can transform any other scan data into avirtual patient model or other virtual and/or parametric representationof the patient's hard tissue of interest in any other way.

4.1 Virtual Patient Model Layers

In one variation shown in FIG. 4, the computer system stores anatomicaland surgical plan data in a set of layers in the virtual patient model.For example, the virtual patient model can include: a first layercontaining a 3D representation of the patient's bone structure aroundthe hard tissue of interest; a second layer containing a 3Drepresentation of the patient's cartilage structure around the hardtissue of interest; a third layer containing a 3D representation of thepatient's musculature and ligature around the hard tissue of interest; afourth layer containing a 3D representation of the patient's skinsurface profile around the hard tissue of interest; a fifth layercontaining a 3D representation of a surgical guide located at a targetposition on the patient's hard tissue of interest prior to resection ofthe hard tissue of interest; a sixth layer containing a 3Drepresentation of the patient's hard tissue of interest followingresection of this hard tissue of interest according to the predefinedsurgical plan; a seventh layer containing a 3D representation of atarget position and orientation of a surgical implant relative to thepatient's hard tissue of interest as specified in the predefinedsurgical plan; etc., such as for each hard tissue of interest (e.g.,both a femur and a tibia) specified for the surgery. As described below,the computer system can then selectively enable and disable these layerspresented on a display in the operating room, such as through awall-mounted display or augmented reality headset worn by a surgeon inthe operating room (or via a virtual reality headset worn by a remotephysician or student).

Therefore, in this implementation, the computer system can: access apre-operative scan of the patient's hard tissue of interest (e.g., afemur and a tibia); extract a three-dimensional contour of the hardtissue of interest from the pre-operative scan; extract athree-dimensional constellation of soft tissue features of the patientfrom the pre-operative scan; compile the three-dimensional contour ofthe hard tissue of interest and the three-dimensional constellation ofsoft tissue features of the patient into the virtual patient model; andstore the virtual patient model—in association with the patient—in adatabase. Later, the computer system can access this virtual patientmodel from the database during the surgical operation on the patient.

In this variation, the computer system can also: track surgicalsteps—such as reorientation of the patient or a portion of the patient,incision into the patient's body, excision of a tissue within thepatient's body, installation of a surgical implant, etc.—throughout thesurgical operation, as described below; and selectively enable anddisable layers of the virtual patient model accordingly.

In one example, the computer system can: register the first layer of thevirtual patient model to the hard tissue of interest of the patientdetected during the subsequent surgery prior to resection of the hardtissue of interest; derive a spatial relationship between features inthe virtual patient model and intermediate features detected on thepatient and near the hard tissue of interest prior to resection of thehard tissue of interest; and then preserve spatial alignment between thevirtual patient model and the patient based on these intermediatefeatures (and any hard tissue of interest features still present)following resection of the hard tissue of interest. The computer systemcan then selectively enable and disable layers in the virtual patientmodel based on current step of the surgical operation, such as by:enabling the first layer exclusively following resection of the hardtissue of interest in order to communicate a difference between theoriginal hard tissue of interest (depicted virtually) and actualresection of the hard tissue of interest visible in the surgical field;enabling the fifth layer exclusively following resection of the hardtissue of interest in order to communicate a difference between thetarget resected profile of the hard tissue of interest (depictedvirtually) defined in the surgical plan and actual resection of the hardtissue of interest visible in the surgical field; and enabling the sixthlayer exclusively following installation of a surgical implant on ornear the hard tissue of interest in order to communicate a differencebetween the target placement of the surgical implant (depictedvirtually) relative to the hard tissue of interest and the actualplacement of the surgical implant on the hard tissue of interest visiblein the surgical field.

5. Optical Scans

Block S120 of the method S100 recites, during a first period of timesucceeding incision of the patient proximal the hard tissue of interestand prior to resection of the hard tissue of interest within a surgicaloperation, accessing a first sequence of optical scans recorded by anoptical sensor facing the surgical field occupied by the patient.Generally, in Block S120, the computer system can interface with one ormore cameras or other sensors to collect optical scan data and/or otherdata representative of a surgical field occupied by the patient, asshown in FIGS. 1A and 2.

In one implementation, the computer system can interface with a singleoptical sensor (e.g., an infrared, LIDAR, depth and/or any other opticalsensor), such as a forward-facing camera arranged on an augmentedreality headset worn by a surgeon within the surgical field. In anotherimplementation, the computer system can interface with an array ofoptical sensors arranged at various locations of the surgical field(e.g., worn by a surgeon, a technician, a nurse, a surgical resident, oran anesthesiologist, or arranged at discrete static locations such asover the surgical field, adjacent a monitor within the surgical field,etc.). In this implementation, the computer system can access opticalscan data from each optical sensor in the array of optical sensors andstitch together the optical scans to generate a three-dimensional (or“3D”) panoramic image of the surgical field. The computer system canthen render the 3D image onto a display, such as a heads-up (or eyes-up)display integrated into an augmented reality headset worn by a surgeon,so that the surgeon may view the 3D image of the surgical field from hernatural perspective within the surgical field and/or from any otherperspective selected by the surgeon (e.g., from the perspective of asurgical resident or technician on an opposite side of the surgicalfield from the surgeon). (The computer system can similar generate andserve virtual reality frames depicting similar content to a virtualreality headset worn by a remote physician or student, such as inreal-time.)

For example, the computer system can download digital photographic colorimages from a forward-facing camera or optical sensor arranged on eachside of an augmented reality headset worn by a surgeon during thesurgical operation. In another example, the computer system can downloaddigital photographic color images from multiple downward-facing camerasarranged in a fixed location over an operating table within an operatingroom. In these examples, the computer system (or a remote computercontracted by the computer system) can stitch optical scans capturedsubstantially simultaneously by two or more cameras within the operatingroom into a 3D point cloud or other 3D image of a volume within theoperating room (hereinafter “3D surgical field image”).

In a similar implementation, the computer system can: access a firstsequence of color images from a fixed stereo camera arranged over andfacing an operating table within the surgical field; transform the firstsequence of color images into a first set of three-dimensional colorpoint clouds; and combine the first set of three-dimensional color pointclouds into a composite three-dimensional color point cloud depictinghard tissue and soft tissue of the patient in Block S120. Based on thecomposite three-dimensional color point cloud, the computer system canthen detect the hard tissue of interest in Block S132 and select the setof intermediate features in the Block S136, as described below.

The computer system can additionally or alternatively download distancedata, such as in the form of a 3D point cloud output by a LIDAR sensorarranged over the operating table. The computer system can further mergedigital photographic color images with distance data to generate asubstantially dimensionally-accurate color map of a volume within theoperating room.

The computer system can collect these optical scan data in Block S120and process these optical scan data as described below substantially inreal-time. The computer system can collect optical scans from one ormore cameras—in fixed locations or mobile within the surgical field—ordistance data from one or more other sensors at a frame rate similar toa projection frame rate of the augmented reality device, such as thirtyframes per second. However, the computer system can collect any othercolor, distance, or additional data from any other type of sensorthroughout a surgery.

5.1 Feature Detection

In one implementation shown in FIGS. 1A and 3, the computer system canimplement edge detection, template matching, and/or other computervision techniques to process the 3D surgical field image to identify ahuman feature (e.g., a skin feature, the hard tissue of interest) in thereal surgical field in Block S140 and can then align the virtual patientmodel to the human feature within the virtual surgical environment. Bythus mapping a virtual patient model within the virtual surgicalenvironment onto real patient tissue identified in the 3D surgical fieldimage, the computer system can later generate an augmented reality framecontaining virtual features aligned to real patient tissue in thesurgical field, such as by projecting the virtual surgical environmentonto the surgeon's known or calculated field of view, as describedbelow.

In one example, the computer system can: transform 2D optical scanscaptured by cameras within the operating room into a 3D surgical fieldimage; identify the patient's left leg in the 3D surgical field image;and map the virtual patient model of the patient's left leg transformedfrom scan data of the patient's left leg onto the patient's left leg inthe 3D surgical field image. In this example, the computer system canimplement object detection, edge detection, surface detection, and/orany other computer vision technique to distinguish distinct volumes orsurfaces in the 3D surgical field image. The computer system can thencompare the virtual patient model to distinct volumes or surfaces in the3D surgical field image to identify the patient's lower left legrepresented in the 3D surgical field image. Similarly, the computersystem can compare the virtual patient model to these distinct volumesor surfaces in the 3D surgical field image to identify the patient'sleft thigh represented in the 3D surgical field image.

In the foregoing implementation, the computer system can compare varioustissue types in the virtual patient model and in the 3D surgical fieldimage to align the virtual patient model to the 3D surgical field image.In particular, the computer system can implement edge detection, colormatching, texture recognition, and/or other computer vision techniquesto distinguish skin, muscle, bone, and other tissue in the 3D surgicalfield image. Therefore, the computer system can: associate a smooth,non-geometric surface with skin; associate a rough red surface insetfrom a skin surface with muscle; and associate a smooth, light pink or(near-) white surface inset from both skin and muscle surfaces as bone.The computer system can then label points or surfaces in the 3D surgicalfield image accordingly. The computer system can therefore detectdifferent types of tissue within the surgical field and dynamically mapthe virtual patient model to one or more tissue types throughout asurgery as the patient's body is manipulated and as different tissuesare exposed.

The computer system can also identify and characterize substantiallyunique tissue features and contours within the patient's scan data. Forexample, for scan data of a patient designated for an upcoming hipsurgery, the computer system can characterize the size and geometry ofthe cotyloid fossa of the patient's acetabulum and then referencesurgical operations on the patient's hip in the surgical plan to theseunique features of the patient's cotyloid fossa. Later, during theoperation, the computer system can: detect such features on thepatient's cotyloid fossa in a feed of images of the surgical field whenthe patient's hip is opened and the cotyloid fossa is exposed; andorient (or align) a virtual patient model of the acetabulum to thecotyloid fossa shown in the optical scan feed. In another example, thecomputer system can access scan data recorded by a multispectral camerain the operating room and distinguish different hard and soft tissues inthe surgical field based on different multispectral signatures of thesetissues; the computer system can then project boundaries of differenttissues identified in these multispectral data onto a concurrent depthimage to isolate and extract 3D geometries of these different hard andsoft tissues from the depth image.

In one variation, the computer system can sequentially detect objectswithin the surgical field according to a hierarchy. For example, thecomputer system can sequentially detect objects in an optical scan ofthe surgical field in the following order: an operating table; thepatient and a hard tissue of interest of the patient; a soft tissuecomponent within the hard tissue of interest of the patient; vascularfeatures of the patient; neuromuscular components; and, finally, a hardtissue of interest (e.g., a bone or subset of bones). Alternatively, thecomputer system can selectively detect objects in the optical scan ofthe surgical field in any order.

Alternatively, the computer system can detect and identify a particularconfirmation gesture performed by the surgeon, nurse, or other humanwithin the surgical field to locate a particular feature of the patientwithin the surgical field. For example, the computer system can detect,in the optical scan of the surgical field or in the field of view of thesurgeon, a gloved hand (e.g., a blue glove) contacting a surface withinthe surgical field. The computer system can then identify the contactwith the surface as confirmation that an overlay frame depicting thevirtual patient model of the patient is properly aligned with thepatient (i.e., the surface the surgeon contact). For example, thecomputer system can identify contact by a gloved hand with a leg asalignment with a correct leg (i.e., a leg the surgeon may prepare forsurgery).

Furthermore, as described above, the computer system can extract rangeof motion and articulation information for anatomical components fromthe virtual patient model; define registration parameters forregistering objects within the surgical field as a hard tissue ofinterest depicted within the virtual patient model; and then locateobjects within the surgical field that conform to the registrationparameters. For example, the computer system can access scan data thatdepicts a three-degree valgus articulation deformity between the tibiaand the femur at a patient's left knee. Then the computer system canscan the surgical field for an object with a three-degree valgusarticulation. The computer system can ignore features and objects withinthe surgical field without a three-degree valgus articulation and, thus,expedite alignment between the virtual patient model and the patient'sleft knee.

However, the computer system can implement any other method or techniqueto detect a surface or volume corresponding to a region of a patient'sbody to align a virtual patient model of the patient to the region of apatient's body in the real surgical environment. The computer system canalso repeat the foregoing process for each optical scan retrieved inBlock S120 substantially in real-time throughout the surgical operation.

6. Pre-Incision: Coarse Registration

In one variation shown in FIGS. 1A and 3, the computer system coarselyregisters the virtual patient model to the hard tissue of interest inthe surgical field based on patient features detected in optical scansprior to the surgeon incising the patient near the hard tissue ofinterest and/or prior to exposure of the hard tissue of interest.

In one implementation, during an initial period of time precedingexposure of the hard tissue of interest within the surgical operation,the computer system: accesses an initial sequence of optical scansrecorded by the optical sensor; detects a head of the patient in theinitial sequence of optical scans; detects a foot in the initialsequence of optical scans; derives an orientation of the patientrelative to the optical sensor based on locations of the patient's headand foot in the initial sequence of optical scans; scans regions of theinitial sequence of optical scans near the detected head for a face oreyes of the patient to determine whether the patient is lying on herfront or back; and predicts a region of the surgical field occupied bythe hard tissue of interest based on the orientation of the patient anda human anatomy model. The computer system then scans the region in thesurgical field depicted in the initial sequence of optical scans for asoft tissue (e.g., skin near the patient's left knee) proximal the hardtissue of interest (e.g., the patient's left femoral condyle and lefttibial plateau). Upon detecting soft tissue features in this region ofthe surgical field, the computer system can then coarsely register thevirtual patient model to these soft tissue features, such as includingorienting the virtual patient model based on the detected orientation ofthe patient (e.g., to set the longitudinal axis of the virtual patientmodel parallel to the longitudinal axis of the patient's torso).

For example, in Block S110, the computer system can generate a virtualscale representation of the patient's left leg, such as in the form of avirtual patient model that includes a dimensionally-accurate contour,surface, and/or volumetric anatomical hard tissue and soft tissuefeatures of the patient's left leg based on an MRI scan of the patient'sleft leg. During a subsequent surgery, the computer system can map thevirtual patient model to real features of the patient's body—detected inoptical scan data (e.g., 2D or 3D color images) recorded by an opticalsensor facing the surgical field—in order to anticipate locations,dimensions, and contours, etc. of both visible and obscured anatomicalfeatures (e.g., a patella, a tibial head, or other sub-dermal tissues)of the patient. In particular, prior to a first incision into thepatient during the surgical operation, the computer system can access avideo feed of a surgical field from an optical sensor arranged overheadthe operating table (or from a camera arranged on an augmented realityheadset worn by a surgeon); detect the operating table, a human body, ahead or face, feet, and a side of the body facing the optical sensor;derive an orientation of the patient's body relative to the camera basedon the position of the head or face and feet; predict a location of thehard tissue of interest (e.g., the patient's left leg) in the field ofview of the optical sensor; and scan this location for a leg. Upondetecting a leg in this location, the computer system can coarselyregister the virtual patient model of the patient's left leg to thedetected leg in the surgical field. The computer system can initiallyrefine this coarse registration by calculating a best fit of a 3D skinsurface contour or envelope represented in the virtual patient model toa contour of the leg detected in the surgical field.

However, the computer system can implement any other method or techniqueto coarsely register the virtual patient model to the patient.

7. Joint Articulation and Mechanical Axis Reconstruction

In one variation shown in FIG. 1A, the computer system calculates amechanical axis of the hard tissue of interest. For example, thecomputer system can: track a constellation of features (e.g., skinfeatures, intermediate features described below) on the patient in asequence of optical scans prior to resection of the hard tissue ofinterest; detect movement of these features within these optical scans;and then derive a real mechanical axis of the hard tissue of interestfrom movement of these features, such as by calculating a best-fit linethat preserves relative positions of features in the constellation overa range of positions of the patient's hard tissue of interest detectedin these optical scans.

In one implementation, the computer system serves a prompt to a surgeonin the surgical field to manipulate a portion of the patient proximalthe hard tissue of interest through a range of motion during a period oftime. As the patients move the portion of the patient (e.g., thepatient's left hip joint, left knee, and left ankle) through this rangeof motion, the computer system can: record a sequence of optical scans;track motion of the patient's upper left leg (e.g., superficial softtissue features on patient's upper left leg) relative to the patient'ship or lower torso in these optical scans; and derive a joint center ofrotation of the patient's left hip relative to the patient's lowertorso. Similarly, as the surgeon articulates the patient's left kneejoint, the computer system can: track motion of patient's upper legrelative to her lower leg; derive a joint center of rotation of thepatient's left knee relative to superficial soft tissue features on thepatient's left leg; and/or derive a mechanical axis of the patient'sleft femur, such as by calculating a line—referenced to the soft tissuefeatures of the patient (and later to hard tissue features of thepatient)—that intersects both the joint center of rotation of thepatient's left hip and the joint center of rotation of the patient'sleft knee. Furthermore, as the surgeon rotates the patient's left anklejoint, the computer system can track motion of the patient's left footrelative to her left lower leg; derive a joint center of rotation of thepatient's left ankle joint relative to superficial soft tissue featuresthe patient's left leg and left foot; and/or derive a mechanical axis ofthe patient's tibia, such as by calculating a line—referenced to thesoft tissue features of the patient (and later to hard tissue featuresof the patient)—that intersects both the joint center of rotation of thepatient's left knee and the joint center of rotation of the patient'sleft ankle.

Alternatively, the computer system can passively track these features inthe surgical field as the surgeon prepares the patient for incision onthe operating table and then implement the foregoing methods andtechniques to derive mechanical axes of hard tissue in the patient'sleft leg. The computer system can similarly derive a kinematic axis ofrotation of the patient's knee.

The computer system can then further refine course registration of thevirtual patient model to the patient by aligning a virtual mechanicalaxis of the hard tissue of interest—defined in the virtual patientmodel—with the corresponding (real) mechanical axis derived from opticalscan data recorded during the surgery, such as by aligning both: themechanical axis of a virtual femur in the virtual patient model to themechanical axis of the patient's femur thus identified in the surgicalfield; the kinematic axis of a virtual leg in the virtual patient modelto the kinematic axis of the patient's knee thus identified in thesurgical field.

The computer system can implement similar methods and techniques todetect mechanical axes, anatomical axes, and/or kinematic axes of thepatient's tissue of interests, such as: an anatomical axis of thepatient's femur; an anatomical axis of the patient's tibia; a mechanicalaxis of the patient's femur; a mechanical axis of the patient's tibia;and/or a mechanical axis of the patient's femur; and a mechanical orkinematic axis of the patient's leg (e.g., from hip to ankle) for atotal knee replacement surgery. The computer system can then refinecourse registration of the virtual patient model to the patient byaligning a virtual anatomical, mechanical, and/or kinematic axes of thehard tissue of interest—defined in the virtual patient model—with thecorresponding axis derived from optical scan data recorded during thesurgery.

8. Post-Incision: Coarse Registration Confirmation

Furthermore, once the surgeon incises the patient near the hard tissueof interest, the computer system can detect this incision to verifycoarse registration of the virtual patient model.

In one implementation, after incision of the patient proximal the hardtissue of interest, the computer system can: continue to access orrecord optical scans of the surgical field; detect presence of a redsurface (e.g., red pixels, which may depict blood or muscle tissue) inthis sequence of optical scans; and confirm registration of the virtualpatient model to soft tissue features proximal the unexposed hard tissueof interest if the location of this detected red surface intersects thevirtual patient model thus coarsely registered to the patient.

In particular, presence of red pixels in the surgical field may indicateblood or muscle tissue near the hard tissue of interest. Therefore, ifthe computer system detects red pixels near the virtual hard tissue ofinterest depicted in the tissue virtual patient modelscoarsely-registered to the patient, the computer system can verify thiscoarse registration of the virtual patient model.

9. Post-Incision: Hard Tissue of Interest Features

Blocks S130, S132, and S134 recite: accessing a first sequence ofoptical scans recorded by an optical sensor facing a surgical fieldoccupied by the patient; detecting a first contour of the hard tissue ofinterest in the first sequence of optical scans; and registering virtualhard tissue features defined in the virtual patient model to the firstcontour of the hard tissue of interest, respectively. Generally, inBlocks S130, S132, and S134, the computer system can: detect theunresected contour of the hard tissue of interest—once exposed by thesurgeon following incision into nearby soft tissue—in a sequence ofoptical scans of the surgical field; and then refine registration of thevirtual patient model to the patient by aligning virtual hard tissue ofinterest features defined in the virtual patient model to real hardtissue of interest features detected in these optical scans. Inparticular, in Blocks S130, S132, and S134, the computer system canrefine coarse registration of the virtual patient model to the hardtissue of interest, such as based on alignment of virtual hard tissuefeatures defined in the virtual patient model and an unresected contourof the hard tissue of interest detected in the surgical field, as shownin FIGS. 1A, 2, and 3.

In one implementation, the computer system can scan a region in asequence of optical scans that intersects the coarsely-registeredvirtual patient model for exposed hard tissue (e.g., a real unresectedcontour of a real femoral condyle), such as depicted by white pixels(e.g., bone) surrounded by red pixels in these optical scans. When ahard tissue surface is thus detected, the computer system can: extract a3D surface profile of the exposed hard tissue from this sequence ofoptical scans, such as by implementing methods and techniques describedabove and below; confirm that this 3D surface profile approximates ageometry of a virtual hard tissue of interest defined in the virtualpatient model (e.g., either a tibial plateau or a femoral condyle); and,if so, extract representative features from this 3D surface profile ofthe exposed hard tissue of interest. The computer system can then: matchthese representative features of the real hard tissue of interest tolike virtual hard tissue of interest features defined in the virtualpatient model (e.g., a virtual unresected contour of a virtual femoralcondyle derived from a pre-operative scan of the patient's knee andstored in the virtual patient model); and snap these virtual hard tissueof interest features to their corresponding real hard tissue of interestfeatures in order to register the virtual patient model to the patient.

In a similar implementation, the computer system can: derive the actualmechanical axis of the hard tissue of interest (e.g., the mechanicalaxis of the patient's femur) thus detected in the surgical field, asdescribed above; align (or “snap”) a virtual mechanical axis of the hardtissue of interest defined in the virtual patient model to the actualmechanical axis of the hard tissue of interest detected in the surgicalfield, thereby virtually constraining the virtual patient model to thepatient in four degrees of freedom; and then snap virtual hard tissuefeatures defined in the virtual patient model to the unresected contourof the hard tissue of interest detected in the surgical field. Inparticular, the computer system can translate and rotate the virtualpatient model to a position relative to the hard tissue of interestdetected in the surgical field that minimizes error (e.g., offset)between virtual hard tissue of interest features in the virtual patientmodel and corresponding real hard tissue of interest features detectedin the surgical field, thereby constraining the virtual patient model tothe patient in six total degrees of freedom.

9.1 Example: Femur

In one example implementation, the virtual patient model includes avirtual representation of the patient's unresected femur, which definesa hard tissue of interest for the surgery. In this exampleimplementation, the computer system can: detect an unresected contour ofa femoral condyle of the patient in the current sequence of opticalscans; and then register virtual unresected femoral condyle featuresdefined in the virtual patient model to the unresected contour of thefemoral condyle detected in this sequence of optical scans.

For example, the computer system can: detect exposed bone near acoarsely-registered virtual patient model of the patent's left leg in asequence of optical scans; extract a 3D surface profile of this exposedbone from these optical scans; and identify this exposed bone as lateraland medial femoral condyles, such as based on similarity between this 3Dsurface profile and a generic femoral condyle model or similaritybetween this 3D surface profile and virtual femoral condyles representedin the virtual patient model. The computer system can then snap avirtual 3D surface profile (or constellation of femoral condylefeatures) of the femoral condyle represented in the virtual patientmodel to the 3D surface profile of the exposed bone (or constellation offeatures representative of the exposed bone) in order to refinealignment and minimize error between the virtual femur in the virtualpatient model and the real hard tissue of interest in the surgicalfield.

In this example, the computer system can also verify alignment betweenthe mechanical axis of the virtual femur depicted in the virtual patientmodel and the real mechanical exit derived from motion of the patient'sleft leg during the surgery, as described above.

9.2 Example: Tibia

The computer system can implement similar methods and techniques toregister a virtual representation of the patient's unresected tibia toanother exposed bone surface in the surgical field. In this exampleimplementation, the virtual patient model can also include a virtualrepresentation of the patient's unresected tibia, which defines a secondhard tissue of interest for the surgery. The computer system cantherefore: detect an unresected contour of a tibial plateau of thepatient in the current sequence of optical scans; and then registervirtual unresected tibial plateau features defined in the virtualpatient model to the unresected contour of the tibial plateau detectedin this sequence of optical scans.

For example, the computer system can: detect a second exposed bone neara coarsely-registered virtual patient model of the patent's left leg inthe same sequence of optical scans described above; extract a 3D surfaceprofile of this second exposed bone from these optical scans; andidentify this exposed bone as a tibial plateau, such as based onsimilarity between this 3D surface profile and a generic tibial plateaumodel or similarity between this 3D surface profile and a virtual tibialplateau represented in the virtual patient model. The computer systemcan then snap a virtual 3D surface profile (or constellation of femoralcondyle features) of the tibial plateau represented in the virtualpatient model to the 3D surface profile of the second exposed bone (orconstellation of features representative of the second exposed bone) inorder to refine alignment and minimize error between the virtual tibiain the virtual patient model and this second hard tissue of interest inthe surgical field.

The computer system can therefore: register a virtual femur in thevirtual patient model to a femoral condyle detected in the surgicalfield; separately register a virtual tibia in the virtual patient modelto a tibial plateau detected in the surgical field; and virtuallyarticulate the virtual tibia relative to the virtual femur in thevirtual patient model responsive to real changes in angular position ofthe patient's lower leg relative to the patient's upper leg.

9.3 Virtual Patient Model Correction

In one variation, the computer system modifies the virtual patient modelin order to further minimize or eliminate error between a virtualcontour of the hard tissue of interest represented in the virtualpatient model and the actual contour of the hard tissue of interestdetected in the surgical field. In particular, prior to resection of thehard tissue of interest, the computer system can interpret the actualhard tissue of interest detected in the surgical field as a “groundtruth” of the patient's original tissue and then drive the virtualpatient model into alignment with this ground truth.

In one implementation, the computer system: calculates a best-fitlocation of the virtual patient model, relative to the hard tissue ofinterest, that minimizes error between virtual hard tissue featuresdefined in the virtual patient model and the unresected contour of thehard tissue of interest detected in a sequence of optical scans; andthen displaces these virtual hard tissue features defined in the virtualpatient model into alignment with the unresected contour of the hardtissue of interest detected in these optical scans. Similarly, thecomputer system can: extract a 3D surface profile of the exposed,unresected hard tissue of interest from optical scans of the surgicalfield; align the virtual patient model to the patient such that errorbetween a virtual unresected contour of this hard tissue of interest inthe virtual patient model and the 3D surface profile of the exposed,unresected hard tissue of interest is minimized (and such that errorbetween virtual and derived mechanical axes of the hard tissue ofinterest is minimized); and then deform the virtual unresected contourof this hard tissue of interest in the virtual patient model into 3Dsuperficial alignment with the 3D surface profile of the exposed,unresected hard tissue of interest detected in the surgical field.

Therefore, the computer system can detect differences between hardtissue contours detected in the surgical field and like contoursdepicted in the virtual patient model and then adjust the virtualpatient model to reflect these hard tissue contours detected in thesurgical field. For example, the virtual patient model can include avirtual femur defined by a set of perpendicular 3D contour lines. Thecomputer system can thus implement methods and techniques describedabove to calculate a best fit location of the virtual femur in thevirtual patient model that minimizes distances from vertices of theseperpendicular 3D contour lines to the 3D surface profile of the femoralcondyle detected in the surgical field. The computer system can thenadjust (or “snap”) these vertices at intersections of these 3D contourlines defining the virtual femur onto the 3D surface profile of femoralcondyles detected in the surgical field.

In this foregoing implementation, the computer system can: characterizethe deformation of the virtual unresected contour of the hard tissue ofinterest in the virtual patient model that aligns this virtualunresected contour to the actual unresected contour of the hard tissueof interest features detected in the surgical field; and then apply thisdeformation to other virtual hard tissue of interest representations inthe virtual patient model, such as: a virtual target resected contour ofthe hard tissue of interest in the virtual patient model; and a virtualrepresentation of a target position of a surgical implant on the hardtissue of interest in the virtual patient model. Therefore, the virtualpatient model can include multiple layers of representations of varioussteps of the surgery, as described above; and the computer system candeform each of these layers into alignment with the actual unresectedcontour of the hard tissue of interest detected in the surgical field.

9.4 Generic Virtual Anatomical Model

In a similar variation, the virtual patient model includes a virtualanatomical model containing a generic representation of the hard tissueof interest. In this variation, the computer system can implementsimilar methods and techniques to calculate a best-fit location of thevirtual anatomical model, relative to the hard tissue of interest, thatminimizes error between virtual hard tissue features defined in thevirtual anatomical model and the unresected contour of the hard tissueof interest detected in the first sequence of optical scans. Thecomputer system can then deform (or morph) the generic representation ofthe hard tissue of interest into conformity with the unique anatomy ofthe patient by displacing virtual hard tissue features defined in thevirtual anatomical model into alignment with the unresected contour ofthe hard tissue of interest detected in the first sequence of opticalscans.

Therefore, in this variation, the computer system can register a genericvirtual anatomical model to the patient and virtually deform thisgeneric virtual anatomical model into alignment with hard tissue ofinterest features detected in the surgical field, thereby generating avirtual patient model unique to the patient prior to resection of thehard tissue of interest during the surgery, such as if no pre-operativescan of the patient's hard tissue of interest is available.

9.5 Ad Hoc Surgical Plan

In another variation, the computer system can define a target resectedcontour for the hard tissue of interest during the surgery, such asafter the hard tissue of interest is exposed and before the hard tissueof interest is resected (and once a generic virtual anatomical model isaligned to the patient's unique anatomy, as described above). Forexample, once the generic virtual anatomical model is aligned to thepatient's unique anatomy, the computer system can receive a command fromthe surgeon specifying a set of target resection parameters for the hardtissue of interest, such as a sequence of quantitative resectionparameters—for a type of the surgery—spoken by the surgeon orally orentered manually into a touchscreen, touchpad, or other user interfacein or near the surgical field. The computer system can then: projectthis set of target resection parameters onto the virtual representationof the unresected contour of the hard tissue of interest to define atarget resected contour of the hard tissue of interest; and then storethe target resected contour of the hard tissue of interest in thevirtual patient model.

Therefore, the computer system can ingest target resection parametersfor the surgery and then generate a virtual representation of the targetresected contour of the hard tissue of interest accordingly in real-timeduring the surgery.

Similarly, the computer system can: ingest commands for position of asurgical implant on the hard tissue of interest; project a virtualrepresentation of the surgical implant onto the virtual patient modelaccording to these commands to define a virtual representation of thehard tissue of interest with implant; subtract a virtual volume of thissurgical implant from the virtual unresected contour of the hard tissueof interest to generate a virtual representation of the target resectedcontour of the hard tissue of interest; and then store both the virtualrepresentation of the hard tissue of interest with implant and thevirtual representation of the target resected contour of the hard tissueof interest.

10. Post-Incision: Intermediate Features

Block S136 of the method S100 recites detecting a set of intermediatefeatures, on the patient and proximal the hard tissue of interest, inthe first sequence of optical scans; and Block S140 of the method S100recites deriving a spatial relationship between the set of intermediatefeatures and the virtual patient model based on registration of thevirtual patient model to the hard tissue of interest. Generally, inBlock S136, the computer system can identify a set (or “constellation”)of real features on and/or near the hard tissue of interest predicted topersist throughout the surgery (i.e., predicted to not be removed fromthe patient during the surgery), as shown in FIGS. 1A and 3. When thevirtual patient model registers to hard tissue of interest in BlockS134, the computer system can then calculate a spatial relationshipbetween the virtual patient model and this set of real, persistentfeatures (or “intermediate features”) in Block S140. Later, as the hardtissue of interest is modified (e.g., resected, cut connected to asurgical implant) during the surgery, the computer system can transitionto registering the virtual patient model to the hard tissue of interestvia this set of real, persistent features rather than directly to thereal hard tissue of interest.

For example, in Block S136, the computer system can aggregate a set ofintermediate features that includes a constellation of visible skinfeatures on the patient proximal the hard tissue of interest, such as:moles; freckles; bruises; veins; or notes or fiducials applied bymedical staff with an ink marker. The computer system can also includethe mechanical axis of the hard tissue of interest in this set ofintermediate features and/or a surface profile or contour of thepatient's skin near and offset from the exposed hard tissue of interest.In one variation, the computer system can also incorporate a 3D geometryof the resected contour of the tissue of interest in this set ofintermediate features and leverage the resected contour to register thevirtual patient model the patient's anatomy throughout the surgery, suchas until the resected contour of the tissue of interest is obscured byan artificial component or again resected (at which time the computersystem can update the set of intermediate features to reflect thisanatomical change).

Furthermore, the computer system can project a virtual target resectedcontour of the hard tissue of interest onto the hard tissue of interestdetected in the surgical field to identify a secondary surface on thehard tissue of interest predicted to remain unchanged during thesurgery, extract bone features from this secondary surface, and appendthe set of intermediate features with these bone features. The computersystem can also compile intermediate features that span all or a largesegment of the circumference of the patient's appendage containing thehard tissue of interest, such as between a minimum distance (e.g., 20centimeters) and a maximum distance (e.g., 50 centimeters) from the hardtissue of interest.

The computer system can then store a 3D spatial map of theseintermediate features relative to the virtual patient model when thevirtual patient model is registered to the hard tissue of interest. Forexample, the computer system can: generate a 3D map of the constellationof intermediate features detected in the surgical field; define anintermediate origin to this 3D map; assign a model origin to the virtualpatient model; calculate a transform or quaternion that represents anoffset between the intermediate origin and the model origin when thevirtual patient model is registered to the patient's real hard tissue ofinterest; and store this transform or quaternion as the spatialrelationship between these intermediate features, the hard tissue ofinterest, and the virtual patient model.

The computer system can implement this process for each hard tissue ofinterest specified in the virtual patient model. For example, during atotal knee replacement surgery, the computer system can: define a firstset of intermediate features and derive a first spatial relationshipbetween a virtual femur model and the patient's (real, physical) femur;and similarly define a second set of intermediate features and derive asecond spatial between a virtual tibia model and the patient's (real,physical) tibia.

Finally, once the computer system has registered the virtual patientmodel to the hard tissue of interest in the surgical field and defined aset of intermediate features and a spatial relationship that maps thevirtual patient model to the hard tissue of interest, the computersystem can serve confirmation of registration of the virtual patientmodel and patient features to the surgeon and then prompt the surgeon toexecute a next step of the surgery (e.g., resection of the hard tissueof interest).

11. Registration Refinement Prior to Bone Resection

In one variation, the computer system repeats the foregoing methods andtechniques throughout the surgery and prior to resection of the hardtissue of interest in order to collect additional patient tissue dataand to compile these patient tissue data into a high-resolution,high-accuracy 3D representation of the patient's hard and soft tissuearound the hard tissue of interest.

For example, during a scan cycle, the computer system can: record afirst depth image or first stereoscopic color image via the opticalsensor; detect the patient in a segment of the first image; and generatean initial 3D field representation of the patient based on soft tissuedata contained in the segment of the first image. During a next scancycle, the computer system can: record a second depth image or secondstereoscopic color image; detect the patient in a segment of the secondimage; and augment the 3D field representation of the patient with softtissue data contained in the segment of the second image. The computersystem can repeat this process during a next sequence of scan cycles torefine the 3D field representation of the patient prior to incision nearthe hard tissue of interest. During a third, later scan cycle, thecomputer system can: detect incision of the patient based on presence ofred pixels in the third image; detect the patient in a segment of thethird image; and repeat the foregoing process to further augment the 3Dfield representation of the patient with soft tissue data contained inthe segment of the third image (outside of the red region in the thirdimage). The computer system can repeat this process during a nextsequence of scan cycles to further refine the 3D field representation ofthe patient following incision and prior to resection of the hard tissueof interest. During a fourth, later scan cycle, the computer system can:detect the hard tissue of interest (e.g., bone, such as a femoralcondyle) based on presence of white pixels in the fourth image; detectthe patient in a segment of the fourth image; repeat the foregoingprocess to augment the 3D field representation of the patient with softtissue data contained in the segment of the fourth image (outside of theexposed bone and red soft tissue area of the fourth image); and injecthard tissue of interest data—depicting the location, orientation, andgeometry of a bone surface (e.g., a femoral condyle, a tibial plateau)detected in the fourth image—into the 3D field representation of thepatient such that these hard tissue of interest data are referenced tosoft tissue features (and/or vice versa) in the 3D field representationof the patient. In this example, the computer system can repeat thisprocess during a next sequence of scan cycles in order to further refinethe 3D field representation of the patient, including both the patient'ssoft and hard tissue and references therebetween.

The computer system can therefore compile anatomical patient dataextracted from a series of scan cycles into a more complete and accurate(e.g., low noise, high-fidelity) 3D virtual representation of thepatient's hard and soft tissue around the hard tissue of interest byover a series of scan cycles.

The computer system can then implement methods and techniques describedabove to: register the virtual patient model to hard tissue depicted inthis 3D field representation of the patient; select the set ofintermediate features from the 3D field representation of the patient;and derive a spatial relationship between these intermediate featuresand the virtual patient model from this 3D field representation of thepatient.

12. Resection

Once the virtual patient model is registered to the patient's hardtissue of interest and intermediate features, the surgeon may execute anext step of the surgery, such as by resecting a portion of thepatient's exposed femoral condyle or tibial plateau according thesurgical plan, as shown in FIGS. 1B and 2.

For example, the surgeon may install a physical guide on the patient andthen manipulate a surgical tool along the physical guide to resect thepatient's femoral condyle. In this example, the virtual patient modelcan include a layer defining a virtual target position of the surgicalguide relative to the hard tissue of interest. The computer system cantherefore implement methods and techniques described below and in U.S.patent application Ser. No. 15/594,623 to generate an augmented realityframe depicting a target location of the surgical guide—defined in thesurgical plan—aligned to the hard tissue of interest in the surgeon'sfield of view of the surgical field. An augmented reality headset wornby the surgeon can then render this augmented reality frame in order tovisually guide the surgeon in placing the surgical guide on the patient.Similarly, the computer system can implement methods and techniquesdescribed below to detect a difference between actual placement of thesurgical guide and the target position of the surgical guide and promptthe user to make adjustments to the position of the surgical guideaccordingly.

In another example, the virtual patient model includes a layer defininga target resected contour of the hard tissue of interest in the form ofa virtual 3D representation of the hard tissue of interest. The computersystem can therefore implement methods and techniques described belowand in U.S. patent application Ser. No. 15/594,623 to generate anaugmented reality frame depicting the target resected contour of thehard tissue of interest is aligned to the hard tissue of interest in thesurgeon's field of view of the surgical field. An augmented realityheadset worn by the surgeon can then render this augmented reality framein order to visually guide the surgeon in either: placing the surgicalguide on the patient; or manipulating a surgical tool along theaugmented reality depiction of the target resected contour of the hardtissue of interest without a physical surgical guide in the surgicalfield.

Furthermore, the computer system can aggregate resected contour data(and surgical implant position data) collected during a surgery andsimilarly serve these data to a remote physician portal to enable aremote physician to track progress of the surgery and to returnrecommendations or prompts to the surgeon currently operating on thepatient. For example, the computer system can generate augmented realityor virtual reality frames depicting the surgical field, such as bothreal tissues of interest and virtual representations of these hardtissues of interest, and then serve these to a computing device worn oraccessed by a remote physician to enable the remote physician to“experience” the surgery, such as in (near) real-time. For example, thecomputer system can: select a frame in a sequence of optical scans;project a virtual representation of an unresected contour of the hardtissue of interest, defined in the virtual patient model, onto theframe; write a spatial difference between the actual resected contourand unresected contour (or target resected contour, etc.) to the frame;and then serve the frame to a physician portal—affiliated with a secondsurgeon located remotely from the surgical field—for remote monitoringof the surgery. The computer system can also enable the remote physicianto control or alter parameters of the surgery based on deviations fromthe patient's preoperative anatomical state and/or deviations from thesurgical plan—as depicted in these augmented or virtual reality framesserved to the remote physician—such as by: setting virtual surgicalstops; repositioning virtual objects (e.g., target positions of virtualartificial components, target resected contours) in the surgical plan;enabling or gating subsequent steps of the surgery; or controllingstep-wise robotic execution of the surgical plan.

As the surgeon completes sequential steps of the surgical plan, thecomputer system can: preserve registration of the virtual patient modelto the patients; and selectively activate (e.g., render) and deactivate(e.g., hide) layers of the virtual patient model according to thecurrent step of the surgical plan, such as automatically based onobjects and surfaces detected by the computer system in the surgicalfield or responsive to explicit input from the surgeon.

13. Post-Resection: Registration

Blocks S150, S156, and S154 of the method S100 recite, during a secondperiod of time succeeding resection of the hard tissue of interestwithin the surgical operation: accessing a second sequence of opticalscans recorded by the optical sensor; detecting the set of intermediatefeatures in the second sequence of optical scans; and registering thevirtual patient model to the hard tissue of interest based on thespatial relationship and the set of intermediate features detected inthe second sequence of optical scans.

Generally, during the surgical operation, the surgeon may reorient,relocate, and/or modify a contour (or “surface,” “surface profile”) ofthe hard tissue of interest or other anatomical component within thesurgical field. In particular, visible features on the hard tissue ofinterest with which the computer system initially registered the virtualpatient model to the patient's anatomy may change contour, dimensions,and/or be removed entirely during the surgical operation. Therefore, thecomputer system can implement Blocks S156 and S154 to preserveregistration of the virtual patient model to the patient's anatomy vianearby intermediate features—such as soft tissue, the mechanical axis ofthe hard tissue of interest, and/or features on the hard tissue ofinterest but offset from the resected contour on the hard tissue ofinterest—that have not substantively changed following resection of thehard tissue of interest (such as other than deformation of soft tissuedue to movement, gravity, and other applied strains, such as surgicaltools placed on the patient).

In one implementation, the computer system tracks actual 3D contours ofexposed hard tissues in a subsequent sequence of optical scans andcompares these 3D contours to virtual unresected contours of thecorresponding tissues of interest defined in the virtual patient model.When the computer system detects a difference between the actual 3Dcontour of an exposed hard tissue in the surgical field and the virtualunresected contours of the corresponding hard tissue of interest definedin the virtual patient model, the computer system can transition toregistering the virtual patient model to the patient based on positionsof the intermediate features detected in the surgical field and thestored spatial relationship between these intermediate features and thevirtual patient model.

Once the intermediate features are selected by the computer system, thecomputer system can continue to track these intermediate features in thesurgical field, such as by implementing 3D object tracking to trackthese intermediate features in subsequent optical scans, and then locatethe virtual patient model relative to these intermediate features basedon the stored spatial relationship for this hard tissue of interest. Forexample, during a particular scan cycle, the computer system can: recordan optical scan; detect at least a subset of the intermediate featuresin the optical scan; calculate a 3D position of the intermediate originfor these detected intermediate features during this scan cycle;implement the stored spatial relationship between these intermediatefeatures and the virtual patient model for this hard tissue of interestto calculate a model origin of the virtual patient model relative tothese intermediate features; and then project the virtual patient modelonto this model origin, thereby registering the virtual patient model tothe patient's hard tissue of interest via these intermediate features(e.g., soft tissue features). The computer system can repeat thisprocess throughout the surgery to preserve registration of the virtualpatient model to the patient's hard tissue of interest, as shown in FIG.1B.

Therefore, the computer system can: track visible skin features—such asin addition to other features in the constellation of intermediatefeatures defined in Block S140—in a sequence of optical scans followingresection of the hard tissue of interest; and then regularly realign thevirtual patient model to the hard tissue of interest based onthree-dimensional positions of visible skin features detected in opticalscans of the surgical field and based on the stored spatial relationshipbetween the virtual patient model and these visible skin features.

13.1 Soft Tissue Deformation

In one variation, the computer system implements a gravity-based modelfor soft tissue (e.g., skin, muscle) to predict deformation of softtissue features contained in the set of intermediate features based onchanges in position and orientation of the patient during the surgicaloperation (e.g., the patient's upper and lower leg during a total kneereplacement surgery). In particular, in this variation, the computersystem can: detect an orientation of the hard tissue of interestrelative to gravity; deform the constellation of soft tissue features(e.g., visible skin features)—in the set of intermediatefeatures—according to a soft tissue gravity model based on theorientation of the hard tissue of interest relative to gravity; and thenapply the stored spatial relationship between these intermediatefeatures to the virtual patient model to register the virtual patientmodel to the patient via this deformed constellation of intermediatetissue features.

For example, the virtual patient model can include a soft tissue layerthat represents the patient's skin and muscle tissue around the hardtissue of interest, as described below; and the computer system canpopulate or annotate this soft tissue layer of the virtual patient modelwith soft tissue features contained in the set of intermediate features.During the surgery, the computer system can: track the position andorientation of the hard tissue of interest in the surgical field;implement the gravity-based model for soft tissue to deform the softtissue layer—including representation of the intermediate features—inthe virtual patient model according to the position and orientation ofthe hard tissue of interest relative to gravity; extract a revisedspatial relationship between virtual representations of theseintermediate features and the hard tissue of interest in the virtualpatient model; and then register the virtual representation of the hardtissue of interest in the virtual patient model to the real hard tissueof interest based on this revised spatial relationship.

In this variation, the computer system can implement a fixedgravity-based soft tissue deformation model. Alternatively, the computersystem can generate a custom soft tissue deformation model to predictdeformation of soft tissue around the hard tissue of interest as afunction of position and orientation relative to gravity, such as basedon changes in 3D skin surface geometry of the patient's soft tissuedetected in a sequence of optical scans recorded by the optical sensorbefore, during, and after incision of the knee and prior to resection ofthe hard tissue of interest.

Therefore, the computer system can predict changes in spatialrelationships between soft tissue features—in the constellation ofintermediate features—relative to the hard tissue of interest as afunction of position and orientation of the hard tissue of interest. Thecomputer system can then implement these gravity-corrected spatialrelationships to preserve registration of the virtual hard tissue ofinterest defined in the virtual patient model to the patient's real hardtissue of interest throughout the surgical operation.

14. Spatial Differences

Block S152 of the method S100 recites, during the second period of timesucceeding resection of the hard tissue of interest within the surgicaloperation, detecting a second contour of the hard tissue of interest inthe second sequence of optical scans; and Block S160 of the method S100recites detecting a spatial difference between virtual hard tissuefeatures defined in the virtual patient model and the resected contourof the hard tissue of interest detected in the second sequence ofoptical scans. Generally, in Blocks S152 and S160, the computer systemcan detect a change in the hard tissue of interest in the surgical field(e.g., resection of the hard tissue of interest) and compare this changeto a virtual representation of the unresected contour defined in thevirtual patient model—and registered to the patient via the set ofintermediate features—in order to calculate quantitative metrics and/orgeometric parameters describing the change in the hard tissue ofinterest from its original geometry, as shown in FIGS. 1B, 2, and 3. Bythen presenting these quantitative metrics and/or geometric parameters,the computer system can enable the surgeon to quickly ascertain theabsolute magnitude and geometry of a change in the hard tissue ofinterest from its original state.

14.1 Absolute Resection Characteristics

In one implementation, the computer system implements methods andtechniques described above (e.g., 3D object tracking) to track the hardtissue of interest in optical scans of the surgical field throughout thesurgery. For each optical scan (or set of optical scans), the computersystem can also: extract a 3D surface profile of the hard tissue ofinterest from the optical scan; register the virtual patient model tothe patient via the constellation of intermediate features; and thencalculate a 3D volumetric disjoint between the virtual unresectedcontour of the hard tissue of interest defined in the virtual patientmodel and the 3D surface profile of the hard tissue of interest derivedfrom the optical scan. The computer system can then characterize this 3Dvolumetric disjoint, such as by: storing a maximum thickness of the 3Dvolumetric disjoint as a resection magnitude for the hard tissue ofinterest; calculating an orientation of a longitudinal axis of the 3Dvolumetric disjoint relative to a longitudinal axis or primary axis ofthe hard tissue of interest; or characterizing a flatness ofconcentricity, etc. of a resected contour represented by the 3Dvolumetric disjoint; etc.

In one example in which the virtual patient model defines a femur as ahard tissue of interest, the computer system can: detect a femoralcondyle in an optical scan; extract the actual resected surface of thefemoral condyle from the optical scan, such as in the form of a virtual3D contour or surface flow of the resected surface of the femoralcondyle; and detect a spatial difference between virtual unresectedfemoral condyle features defined in the virtual patient model—aligned tothe patient via the set of intermediate features—and this virtualrepresentation of the actual resected surface of the femoral condyleextracted from the optical scan. In particular, in this example, thecomputer system can: calculate a magnitude of resection of the femoralcondyle based on the spatial difference; calculate an orientation of theresected surface of the femoral condyle based on the spatial difference;and/or characterize a surface profile or form of the resected contour ofthe femoral condyle, such as concentricity, flatness, angularity,symmetry, or position relative to a reference on the femur (e.g., themechanical axis of the femur).

In this example, the virtual patient model can define a tibia as asecond hard tissue of interest. The computer system can therefore:detect a tibial plateau in the same or other optical scan; extract theactual resected surface of the tibial plateau from the optical scan; anddetect a spatial difference between virtual unresected tibial plateaufeatures defined in the virtual patient model—aligned to the patient viathe set of intermediate features—and this virtual representation of theactual resected surface of the tibial plateau extracted from the opticalscan. In particular, in this example, the computer system can: calculatea magnitude of resection of the tibial plateau based on the spatialdifference; calculate an orientation of the resected surface of thetibial plateau based on the spatial difference; and/or characterize asurface profile or form of the resected contour of the tibial plateau,such as concentricity, flatness, angularity, symmetry, or positionrelative to a reference on the femur (e.g., the mechanical axis of thefemur).

In this implementation, the computer system can then visually indicateto the surgeon this absolute difference between the original state ofthe hard tissue of interest and the resected contour of the hard tissueof interest, such as by serving the magnitude of resection, theorientation of resection, and the surface profile to the surgeon. Forexample, the computer system can implement methods and techniquesdescribed above and below to: generate an augmented reality framedepicting the virtual unresected contour of the hard tissue of interestaligned to the actual hard tissue of interest—now resected—in thesurgeon's field of view of the surgical field; and serve this augmentedreality frame to an augmented reality headset worn by the surgeon forrendering in near real-time. In this example, the computer system can:detect a position of an augmented reality headset—worn by asurgeon—proximal the surgical field; estimate a perspective of thesurgeon viewing the surgical field based on the position of theaugmented reality headset in the surgical field; generate an augmentedreality frame that includes a projection of the virtual unresected hardtissue of interest aligned with the real hard tissue of interest of thepatient from the perspective of the surgeon; insert the magnitude ofresection, the orientation of resection, and the surface profile of thehard tissue of interest—derived from the last optical scan of thesurgical field—into an augmented reality frame; and then serve thisaugmented reality frame to the augmented reality headset. The augmentedreality headset can then render the augmented reality frame in nearreal-time.

14.2 Deviation from Target Resection

The computer system can additionally or alternatively compare the actualresected contour of the hard tissue of interest thus detected in thesurgical field to the target resected contour of the hard tissue ofinterest defined in the virtual patient model in order to calculatequantitative metrics and/or geometric parameters describing a differencebetween the actual and target resected contours of the hard tissue ofinterest. By then presenting this difference to the surgeon, thecomputer system can enable the surgeon to quickly ascertain both whetherthe surgeon has deviated from the surgical plan and a magnitude of thisdeviation, such as in six degrees of freedom.

In one implementation shown in FIG. 2, the computer system: extracts anactual resection contour of the hard tissue of interest of the patientfrom an optical scan, as described above; and calculates a spatialdifference between the actual resected contour of the hard tissue ofinterest and the target resected contour defined in the virtual patientmodel. For example, the computer system can: calculate a distancemagnitude difference between the actual resected contour of a femoralcondyle and the target resected contour of a femoral condyle defined inthe virtual patient model, such as in the form of a maximum distance inmillimeters between the actual and target resected contours of thefemoral condyle normal to the target resected contour or parallel to themechanical axis of the femur. The computer system can additionally oralternatively calculate an orientation difference between the actualresected contour of the femoral condyle and the target resected contourof the femoral condyle, such as by calculating a best-fit plane of theactual resected contour of the femoral condyle; and calculating angularoffsets between a target resect plane of the femoral condyle defined bythe virtual patient model and the actual resected contour of the femoralcondyle extracted from the optical scan. The computer system can alsocharacterize a surface profile difference between the actual resectedcontour of the femoral condyle and the target resected contour of thefemoral condyle, such as differences between actual and target surfaceroughness, texture, flatness, and/or symmetry, etc. of the femoralcondyle. The computer system can then present these distance magnitudedifference, orientation difference, and surface profile differencemetrics to the surgeon, such as via an augmented reality headset worn bythe surgeon or via a display present near the surgical field.

14.3 Surgical Implant

Furthermore, once the surgeon has resected each hard tissue of interestto within a specified tolerance or verified a deviation from thesurgical plan, the surgeon may then locate one or more surgical implantson the tissue(s) of interest. The computer system can then implementmethods and techniques similar to those described above to: detect andtrack a surgical implant in the surgical field; calculate an actualposition of the surgical implant relative to its corresponding hardtissue of interest; calculate a spatial difference between the actualposition of the surgical implant relative to the hard tissue of interestand a target position of the surgical implant relative to the hardtissue of interest as defined in the virtual patient model (or otherwisedefined in a surgical plan for the surgery); and then communicate thisspatial difference to the surgeon, such as via augmented reality framesserved to the surgeon's augmented reality headset, as shown in FIG. 3.

For example, during the surgery the surgeon may place an artificialfemoral component over the patient's resected femoral condyle. In thisexample, the virtual patient model can include a layer defining a targetposition of a femoral component relative to the hard tissue of interest.The computer system can therefore implement methods and techniquesdescribed herein and in U.S. patent application Ser. No. 15/594,623 togenerate an augmented reality frame depicting the target location of thefemoral component aligned to the hard tissue of interest in thesurgeon's field of view of the surgical field. An augmented realityheadset worn by the surgeon can then render this augmented reality framein order to visually guide the surgeon in aligning the femoral componentto its target position on the patient's femur. Similarly, the computersystem can implement methods and techniques described below to detect adifference between the actual and target positions of the femoralcomponent on the femur and can then prompt the user to make adjustmentsto the position of the femoral component accordingly prior to fasteningor bonding the femoral component to the resected femoral condyle.

For example, the computer system can: access a sequence of optical scansrecorded by the optical sensor after the surgeon confirms resection ofthe tissue(s) of interest; detect the set of intermediate features inthis sequence of optical scans; register the virtual patient model tothe hard tissue of interest based on the stored spatial relationshiplinking the set of intermediate features to the virtual patient model;and detect the surgical implant in the sequence of optical scans, suchas by implementing template matching or object recognition techniques tomatch features extracted from the optical scans to a virtual surgicalimplant model contained in the virtual patient model or otherwise linkedto the surgery. The computer system can then: calculate an actualposition of the surgical implant relative to the virtual patient modelregistered to the hard tissue of interest based on the spatialrelationship and the set of intermediate features; and calculate aspatial difference between the actual position of the surgical implantand the target position of the surgical implant relative to the hardtissue of interest accordingly. The computer system can then render thisspatial difference on a display near the surgical field or serve anaugmented reality indicating this spatial difference to the surgeon'saugmented reality headset, thereby: guiding the surgeon in aligning thesurgical implant to the target implant location specified in thesurgical plan; and/or enabling the surgeon to intentionally deviate fromthis surgical plan by an indicated quantitative linear distance and/orangular offset.

The computer system can repeat this process for each other surgicalimplant designated for the surgery.

14.4 Cumulative Deviation

In one variation, the computer system can track (or log) each deviationfrom the surgical plan throughout the surgery, such as differencesbetween actual and target resected contours of tissues of interestand/or differences between actual and target surgical implant positionsrelative to corresponding tissues of interest. At each step of thesurgical operation, the computer system can then calculate a currentcumulative deviation (or “error”) throughout the surgery up to thecurrent step of the surgical plan. The computer system can then presentthis cumulative deviation to the surgeon (e.g., via an eyes-up orheads-up display in an AR headset worn by a surgeon) throughout thesurgery.

For example, during a total knee replacement surgery, the computersystem can: predict how a difference between the actual and targetresected contours of the patient's femoral condyle will result in adifference between the actual and target positions of an artificialfemoral component on the patient's femur when later installed during thesurgery based on the actual geometry of the resected femoral condyle anda known geometry of the artificial femoral component; and then presentthis predicted deviation to the surgeon in order to quantitativelycommunicate to the surgeon how the result of this current step mayaffect a future step of the surgery. As the surgeon transitions toresecting the patient's tibial plateau, the computer system cansimilarly: predict how a difference between the actual and targetresected contours of the patient's tibial plateau will result in adifference between the actual and target positions of an artificialtibial component on the patient's tibia when later installed during thesurgery based on the actual geometry of the resected tibial plateau anda known geometry of the artificial tibial component; and then presentthis predicted deviation to the surgeon. The computer system can furthercombine these femoral and tibial deviations to predict a cumulativedeviation at conclusion of the surgery, such as including: a spatialdifference between the pre-operative and post-operative joint centers ofrotation of the patient's knee; a difference between the pre-operativeand post-operative lengths of the user's leg; a difference between thepre-operative and post-operative angular resting position of thepatient's foot relative to the patient's hip; and/or a differencebetween the patient's pre-operative and post-operative gait; etc. Thecomputer system can then present these predicted deviations to thesurgeon in real-time, thereby enabling the surgeon to better comprehend,mitigate, and/or verify such deviations from the surgical that may occurupon conclusion of the surgery.

The computer system can continue to generate and serve such deviationpredictions to the surgeon as the surgeon refines resected contours onthe tissues of interest, places surgical implants on the tissues ofinterest, and fastens these surgical implants onto their correspondingtissues of interest.

In another implementation, during the surgery, the computer system canlog deviations during the surgery in memory and project these priordeviations onto the field of view of the surgeon—aligned with the hardtissue of interest within the surgical field—during surgery. Therefore,the surgeon may, in real-time, play back prior deviations and visualizetheir cumulative effects on the patient's anatomy at present to informimminent placement of implants and/or upcoming surgical steps. Forexample, the computer system can render a frame—projected onto the fieldof view of the surgeon—representing a last cut across a femur performedby the surgeon aligned with the hard tissue of interest in the surgicalfield. The computer system can also insert an image of the hard tissueof interest prior to the last cut, an outline of the hard tissue ofinterest after the last cut, and a virtual guide for a next cut into theframe.

14.5 Deviation Notes and Storage

In one variation shown in FIG. 3, the computer system also interfaceswith the surgeon to verify intent to deviate from the surgical plan—suchas intent to deviate from a target resected contour of a hard tissue ofinterest, intent to deviate from a target position of a surgical implantrelative to the hard tissue of interest, and/or intent to deviate fromtarget relative positions of two adjacent surgical implants—and torecord these deviations from the surgical plans, confirmation of thesurgeon's intent to deviate, and the surgeon's reasons for thesedeviations.

In one implementation, after detecting a spatial difference between anactual and a target resected contour of the hard tissue of interest, thecomputer system prompts the surgeon to confirm her intent to deviatefrom the surgical plan according to the spatial difference. In responseto confirmation of intent to deviate from the surgical plan according tothe spatial difference, the computer system can also prompt the surgeonto provide a reason for this deviation. The computer system can thenrecord a reason spoken orally by the surgeon in real-time during thesurgery or record a text-based response provided by the surgeon (ornurse or other staff nearby) via the user interface. Upon receipt ofthis reason, the computer system can store: a representation of thespatial difference (e.g., in the form of a 3D virtual contour of theresected hard tissue of interest extracted from a recent optical scan);confirmation of the surgeon's intent to deviate from the surgical planaccording to the spatial difference; and the reason for the deviationprovided by the surgeon, such as in a database or in a surgery fileassociated with the patient and surgical operation. In thisimplementation, the computer system can also gate a next step of thesurgery (or guidance for a next step of the surgery) until the surgeoneither: confirms her intent to deviate from the surgical plan andprovides a reason for the deviation; or requests guidance to return tothe surgical plan.

Alternatively, the computer system can store the representation of thespatial difference and confirmation of the surgeon's intent to deviatefrom the surgical plan during the surgery. Upon conclusion of thesurgery, the computer system can retroactively prompt the surgeon toprovide a reason for the deviation (e.g., in post-operative surgerynotes). For example, the computer system can present the representationof the spatial difference between the target and actual resectedcontours of the hard tissue of interest to the surgeon via a physicianportal and prompt the surgeon to annotate the representation with areason for the deviation.

The computer system can implement similar methods and techniques tostore representations, intents, and reasons for deviations from targetresected contours of other tissues of interest, target placement ofsurgical implants relative to these tissues of interest, and/or targetrelative positions of two surgical implants located in the patientduring the surgery.

However, in this variation, if the surgeon requests guidance to returnto the surgical plan, the computer system can implement methods andtechniques described below to guide the surgeon in refining the resectedcontour to correct the deviation, as described below.

14.6 Remote Guidance

Alternatively, in response to detecting such deviation from the surgicalplan, the computer system can: generate virtual reality frames depictingboth real tissue of interest in the surgical field and virtual content(e.g., target resected contours of tissues of interest defined in thevirtual patient model thus registered to the real patient tissue); servethese virtual reality frames to a virtual reality headset worn by aremote surgeon logged into the surgery; and prompt the remote surgeon tosuggest changes to the surgical plan. For example, the computer systemcan prompt the remote surgeon to: move virtual objects (e.g., virtualguides, virtual artificial components) or virtual surfaces (e.g., targetresected contours) in these virtual frames in order to modify orredefine target parameters for this surgery; selectively authorizingnext steps of the surgical plan; and communicating directly with thelocal surgeon—such as through an audio and/or video feed—to discuss andverify changes to the surgical plan.

The computer system can therefore detect deviation from the surgicalplan and automatically prompt a remote surgeon to assist the localsurgeon in real-time during the surgery when such deviations aredetected.

15. Adaptation and Guidance

In one variation shown in FIG. 2, in response to detecting deviationbetween an actual resected contour of the hard tissue of interestextracted from an optical scan and a target resected contour defined inthe virtual patient model, the computer system can: modify subsequentsteps of the surgical plan to account for, adapt to, and/or negatedeviations between actual and target resected contours of the hardtissue of interest; and can update layers of the virtual patient modelto virtually reflect these modifications.

In one implementation, the computer system modifies a subsequent step ofthe surgical plan to correct or counteract a detected deviation. In thisimplementation, the computer system can detect a deviation from thesurgical plan that affects a reference point, a reference angle, areference plane, etc. from which a datum in a subsequent step isreferenced. Therefore, the computer system can modify the particularsubsequent step of the surgical plan to reference a datum defined by adifferent feature, incision, etc.

Alternatively, the computer system can modify another target resectedsurface of one hard tissue of interest (e.g., a tibial plateau) based onthe actual resected contour of a nearby hard tissue of interest (e.g., afemoral condyle). For example and as shown in FIG. 5, the computersystem can access a surgical plan for a hip replacement surgery thatdefines: a first surgical step transecting a femoral head of a femuralong a target cut plane; and a second surgical step boring into thefemur along a mechanical axis (i.e., load bearing axis through the femurparallel direction of gravity) of the femur, a bore of the secondsurgical step at an angle to the target cut plane of the first surgicalstep. In this example, the computer system can detect a deviation in thefirst surgical step in which a real cut plane resulting from completionof the first surgical step in the surgical field is offset from (or skewto) the target cut plane by three degrees. Because the bore of thesecond surgical step is defined relative to the target cut plane of thefirst surgical step, the computer system can adjust the second surgicalstep to locate the bore at a new angle (e.g., smaller angle) relative tothe real cut plane executed in the first surgical step such that thebore aligns with the mechanical axis of the femur despite the deviationin the first surgical step.

In the foregoing example, the computer system can additionally oralternatively define an intermediate step—between the first surgicalstep and the second surgical step—that specifies a re-planing operationto cut the femoral head parallel to the (former) target cut planedefining the first surgical step in order to correct the offset (or“skew”) that resulted during the first surgical step. Followingcompletion of the intermediate step, the computer system can detect adeviation of one centimeter between the target cut plane of the firstsurgical step and an actual cut plane of the femoral head resulting fromthe re-planing operation defined in the intermediate step.

In another example, after detecting a deviation between an actualresected contour of the hard tissue of interest and a target resectedcontour extracted from a first surgical step of a surgical plan (andreceiving confirmation of intent of this deviation from the surgeon),the computer system can determine that the deviation directly affects noother subsequent step of the surgical plan. Therefore, the computersystem can record the deviation as described above and notify thesurgeon of acceptance of the deviation.

Similarly, the computer system can cooperate with a surgeon to modifysubsequent steps of the surgical plan. The computer system can serve aprompt to the surgeon to communicate the deviation, a predicted effectof the deviation on subsequent surgical steps and/or surgical outcome,and a suggested modification to a subsequent step that may correct thisdeviation or lessen compounding effects of this deviation.

However, the computer system can modify, maintain, add, and/or removeany subsequent steps of the surgical plan to adapt to deviations andlimit problems in subsequent steps of the surgery in any other suitableway, such as independently and/or in cooperation with the surgeon.

15.1 Insufficient Resection

In a similar variation shown in FIG. 1B, the computer system prompts theuser to refine a resected contour on a hard tissue of interest inresponse to detecting insufficient material removal from the hard tissueof interest based on the spatial difference. For example, the computersystem can extract a 3D contour of the exposed hard tissue of interestin the surgical field following resection by the surgeon and comparethis 3D contour of the actual resected hard tissue of interest to thetarget resected contour of the hard tissue of interest defined in thevirtual patient model in order to determine whether any portion of theactual resected surface of the hard tissue of interest extends beyond(i.e., falls outside of) the target resected contour of the hard tissueof interest defined. More specifically, the computer system candetermine whether insufficient material has been removed from anyportion of the hard tissue of interest as defined in the surgical plan.If the computer system thus detects that a portion of the resectedsurface of the hard tissue of interest still extends beyond the targetresected contour of the hard tissue of interest, such as beyond athreshold offset or maximum tolerance, then the computer system canserve a prompt to the surgeon to remove additional material from thisregion of the hard tissue of interest. For example, the computer systemcan highlight—via the augmented reality headset worn by thesurgeon—portions of the hard tissue of interest that extend beyond thetarget contour of the hard tissue of interest defined in the virtualpatient model.

The computer system can then repeat the foregoing methods and techniquesto monitor this surface of the hard tissue of interest and to calculatea second spatial difference between the actual resected contour of thehard tissue of interest detected in a next sequence of optical scans andthe target resected contour of the hard tissue of interest representedin the virtual patient model—which is still registered to the hardtissue of interest. The computer system can then: confirm that theactual resected contour falls within a predefined tolerance of thetarget resected contour (e.g., by rendering confirmation on thesurgeon's augmented reality headset); prompt the surgeon to furtherresect the hard tissue of interest if insufficient material has beenremoved from the hard tissue of interest; or respond to excessiveremoval of material from the hard tissue of interest by modifying alater step of the surgery, as described below.

15.2 Excessive Resection

In a similar variation shown in FIG. 1B, the computer system modifies atarget resected contour for a second hard tissue of interest (e.g., atibial plateau)—such as automatically or with guidance from thesurgeon—responsive to detecting excessive material removal from a firsthard tissue of interest (e.g., a femoral condyle) based on the spatialdifference calculated in Block S160. For example and as described above,the computer system can extract a 3D contour of the exposed femoralcondyle in the surgical field following resection by the surgeon andcompare this 3D contour of the actual resected femoral condyle to thetarget resected contour of the femoral condyle defined in the virtualpatient model to determine whether any portion of the target resectedcontour of the femoral condyle defined in the virtual patient modelextends beyond (i.e., falls outside of) the actual resected surface ofthe femoral condyle—that is, if excessive material has been removed fromany portion of the femoral condyle beyond resection defined in thesurgical plan. If the computer system thus detects that a portion of theresected surface of the femoral condyle has been removed beyond thetarget resected contour defined in the virtual patient model—such asbeyond a threshold offset or maximum tolerance defined in the surgicalplan—the computer system can: calculate a best fit plane (or othermating profile defined by an artificial femoral component) of theresected contour; calculate a direction and orientation of the offsetbetween this best fit plane (or other mating profile) and the targetresected contour of the femoral condyle; and offset a target resectedcontour of the adjacent tibial plateau opposite this direction andorientation of the femoral condyle offset. (The computer system cansimilarly modify the target resected contour of the adjacent tibialplateau to compensate for this femoral condyle offset based on a knowninteraction between the artificial femoral component and an artificialtibial component specified for the surgery). In this example, responsiveto removal of excessive material from the patient's femoral condyle, thecomputer system can: automatically shift the target resected contour ofthe patient's tibial plateau to compensate; and/or predict a thickershim between artificial femoral and tibial components installed on thepatient during later steps of the surgery.

In this variation, the computer system can also prompt the surgeon toconfirm or adjust this modification to the target contour of the tibialplateau before updating the virtual patient model accordingly.

In this example, the computer system can repeat the foregoing processesonce the tibial plateau is resected to calculate an offset between theactual and (modified) target resected contours of the tibial plateau.Accordingly, the computer system can: predict a shim thickness and/orwedge geometry that compensates for both spatial differences betweenactual and target resected contours of the femoral condyle and tibialplateau; and serve this prediction to the surgeon, such as to inform thesurgeon of predicted effects of the current states of these tissues ofinterest.

15.3 Outcome Probability

In another variation shown in FIG. 1B, the computer system leverages apatient outcome model—linking absolute and/or relative resected tissuecontours and/or surgical implant positions for a surgery type to patientrecovery, recovery rate, satisfaction, etc.—to predict longer-termeffects of tissue resection and/or surgical implant placement during thesurgery on the patient.

In one implementation, the computer system: calculates an absolutespatial difference between the actual resected contour of the hardtissue of interest detected in an optical scan and an unresected contourof the hard tissue of interest defined in the virtual patient model, asdescribed above; accesses a correlation between outcomes and absolutespatial differences between actual resected contours of the hard tissueof interest and unresected contours of the hard tissue of interestwithin a population of patients subject to instances of the surgicaloperation, such as defined in the patient outcome model; and predict aprobability of successful outcome of the patient (e.g., probability thatthe patient will regain 95% of her range of motion; probability that thepatient will fully recover within six months of the surgery; lowprobability of infection; low probability of a second correctivesurgery; high probability of patient satisfaction) based on the absolutespatial difference and the patient outcome model. Then, if the computersystem predicts a high probability of successful outcome (e.g., aprobability of successful outcome that exceeds a threshold probability)given the current resected state of one or more tissues of interest inthe surgical field, the computer system can prompt the surgeon to moveto a next step of the surgical operation. However, if the computersystem predicts a low probability of successful outcome given thecurrent resected state of one or more tissues of interest in thesurgical field, the computer system can prompt the surgeon to correctthe actual resected contour of the hard tissue of interest, such asaccording to methods and techniques described above to reduce thespatial difference.

The computer system can implement similar methods and techniques topredict probability of a successful outcome for the patient based onabsolute resection of a hard tissue of interest—such as relative to anoriginal state of the hard tissue of interest relative to the mechanicalaxis of the hard tissue of interest—rather than based on a differencebetween the target and actual resected contours of the hard tissue ofinterest.

Furthermore, as the surgeon resects various tissues of interest, refinesthese resected contours, and completes each subsequent step of thesurgery, the computer system can input absolute or relative spatialdifferences between actual and target resected contours for thesetissues of interest in order to: update the predicted probability ofsuccessful outcome for the surgery; and thus inform changes to asubsequent step of the surgery or prompt refinement of a current step ofthe surgery.

Furthermore, the computer system can implement similar methods andtechniques to predict probability of successful outcome for the surgerybased on the absolute or relative positions of surgical implants placedon corresponding tissues of interest during the surgery. For example,after the patient's femoral condyles and tibial plateau are resected bythe surgeon, the surgeon may place an artificial femoral component overthe resected end of the patient's femur. The computer system can then:calculate an absolute position of the artificial femoral component overthe resected femoral condyle detected in an optical scan; access acorrelation between outcomes and absolute positions of instances of aartificial femoral component on resected femoral condyles within apopulation of patients subject to total knee replacement surgeries; andpredict a probability of successful outcome of the patient based theabsolute position of the artificial femoral component on the patient'sresected femoral condyle. In response to the probability of successfuloutcome exceeding a threshold probability, the computer system canprompt the surgeon to fasten the artificial femoral component to thepatient's femur in its current position and/or move to a next step ofthe surgical operation. However, in response to the probability ofsuccessful outcome falling below the threshold probability, the computersystem can prompt the surgeon to adjust the absolute position of theartificial implant on the hard tissue of interest; as the surgeonadjusts the position of the artificial femoral component, the computersystem can track the position of the artificial femoral componentrelative to the hard tissue of interest (e.g., relative to theregistered virtual patient model) and recalculate the probability ofsuccessful outcome for the patient accordingly.

16. Prediction

In one variation, the computer system can access historical surgicaldata (e.g., instances of the virtual representation, records ofsurgeries, deviations from surgical plans, and/or compliance withsurgical plans) to extract trends in deviations from surgical plans forparticular surgeons, particular surgical steps, and/or particularsurgery types. Based on these trends, the computer system can predicttimes, locations, magnitudes, and types of deviations to surgical plansfor future operations and adapt the surgical plans accordingly.

In one implementation, the computer system can access historicalsurgical data (e.g., recorded over a one-month period, over a two-yearperiod, and/or over the entire career of a particular surgeon) from aremote computer system documenting surgeries performed by a particularsurgeon. From the historical data, the computer system can extracttrends in the deviations from surgical plans the particular surgeonexecutes for particular surgical steps, surgery types, patient anatomy,patient demographics, etc. From these trends in deviations for theparticular surgeon, the computer system can adapt surgical plans for allfuture surgeries scheduled to be performed by the surgeon to anticipatewhen the surgeon will deviate, the surgeon's preferred course of actionfollowing the deviation, etc.

For example, the computer system can extract a trend from historicalsurgical data for a particular surgeon indicating a surgeon consistentlydeviates five to ten millimeters from a prescribed target resectedcontour at a first surgical step in a surgical plan. The computer systemcan extract a tolerance range for cuts executed by the surgeon (e.g.,between five and ten millimeters for the first surgical step). Thecomputer system can then modify future surgical plans to acceptdeviations within the tolerance range for the particular surgeon andguide the surgeon to remain within the tolerance range throughoutsurgery. Additionally or alternatively, the computer system cancalculate a cumulative tolerance stackup for the surgery defined as asum of a maximum predicted deviation for all or a subset of steps of thesurgery based on the tolerance range for cuts executed by the surgeon.The computer system can then define an acceptable tolerance window foreach surgical step within which the computer system can acceptdeviations without updating subsequent steps and guide the surgeon toremain within the acceptable tolerance window.

In another example, the computer system can extract a trend fromhistorical surgical data for a particular surgeon indicating a surgeonroutinely elects (i.e., intentionally) to drill into a femur with adrill-bit smaller than that which was recommended in the surgeon'ssurgical plans. The computer system can, therefore, modify futuresurgical plans to incorporate the small drill-bit and preemptively modeleffects (e.g., hole size of the incision by the small drill-bit,duration the small drill-bit is inserted to yield a target hole size,and/or trajectory of the small drill-bit) of drilling into a bone withthe small drill-bit. Furthermore, the computer system can calculateeffects of drilling into a bone with the small drill-bit on subsequentsteps of the surgical plan and adapt the subsequent steps accordingly.For example, the computer system can add additional steps of boring outthe hole with two distinct bores to form a hole of a size sufficient toaccept an artificial hip implant.

In another example, the computer system can extract a trend fromhistorical surgical data for a particular surgeon indicating a surgeonroutinely elects (i.e., intentionally) to ream and broach a femur at aslight angle to a mechanical axis of the femur instead of executing aplanned (target) ream into the femur aligned with the mechanical axis asdefined in the surgical plan. Based on consistent election of the reamat a slight angle to the mechanical axis, the computer system cananticipate the surgeon will elect to execute similar broaches into thefemur at the slight angle to the mechanical axis. Therefore, thecomputer system can adapt the surgeon's surgical plan for hipreplacement surgeries to preemptively define the broach at the slightangle to the mechanical axis. Additionally, the computer system canpreemptively adapt subsequent steps of the hip replacement surgical planto account for a ream and broach at the slight angle to the mechanicalaxis. For example, based on the slight angle, the computer system canvirtually model a position of an artificial hip implant followingimplantation into the femur; from the position of the artificial hipimplant, the computer system can predict the position and angle of atransecting cut across the femoral head (i.e., a step preceding the boreinto the femur).

In another example, the computer system can detect a particular surgeonprefers to cut a tibia one degree varus for patients of a firstdemographic group (e.g., sedentary females) and one degree valgus forpatients of a second demographic group (e.g., active males) based onhistorical surgical data for the particular surgeon. Therefore, thecomputer system can define a first surgical plan for the firstdemographic group to include a one-degree varus target resected contour;additionally, the computer system can calculate a one-degree rotation ofthe tibia resulting from the one-degree varus target resected contourand adapt subsequent steps of the first surgical plan accordingly.Similarly, the computer system can define a second surgical plan for thesecond demographic group to include a one-degree valgus target resectedcontour. The computer system can also calculate a one-degree rotation ofthe tibia resulting from the one degree valgus target resected contourand adapt subsequent steps of the second surgical plan accordingly.

Similarly, the computer system can access historical surgical data(e.g., recorded over a one-month period, over a two-year period, over aten-year period, and/or indefinitely) from a remote computer systemdocumenting surgeries of a particular type (e.g., knee replacementsurgery) performed by a group of surgeons. From the historical data, thecomputer system can extract trends in the deviations from surgical plansthe group of surgeons execute for surgeries of a particular type. Fromthese trends in deviations for the particular surgeon, the computersystem can adapt surgical plans for all future surgeries scheduled to beperformed by each surgeon in the group of surgeons to anticipate whenthe surgeons will deviate from the surgical plans and preemptively adaptthe surgical plans to accommodate preferences of the group of surgeonsfor each step of the surgical plans.

However, the computer system can apply historical surgical deviationdata to inform surgical plan definitions in any other suitable way.

17. Deviation and Patient Outcome Model

Furthermore, by tracking and recording deviations as described above,the computer system can correlate surgical deviations with patientoutcomes (e.g., restoration of range of motion, reduction of painlevels, improved levels of function and/or mobility, increase inactivity level, and/or high patient satisfaction scores) to informfuture surgical practices and surgical plans. In particular, thecomputer system can maintain a record of actual resected contour of thetissues of interest, surgical deviations, and surgical outcomes for aplurality of surgeries; extract trends from the historical surgicaldata; and apply these trends to inform surgical plans, outcomes, andacceptable deviations for future surgeries.

In one implementation, the computer system can access patient outcomedata from a remote database. In this implementation, patients and/ormedical staff may manually enter into the remote database (i.e., througha user portal rendered on a display of a computing device) patientoutcome data, such as pain levels, patient satisfaction surveys, levelsof mobility, activity level, etc. Alternatively, a patient's computingdevice (e.g., smartphone) can automatically upload (or push) patientactivity data (e.g., pedometer readouts, heartrate, etc.) to the remotedatabase, such as over a wireless network.

For example, the computer system can detect that a particular surgeonroutinely cuts femurs one degree varus and patients of the particularsurgeon typically exhibit poor outcomes (e.g., high pain levels, limitedrange of motion, and/or low recorded activity levels post-surgery).Therefore, the computer system can modify surgical plans for theparticular surgeon with additional or more detailed virtual guides toguide the surgeon to avoid cutting femurs one degree varus.Additionally, the computer system can adapt (or populate) surgical plansto guide other surgeons to avoid cutting femurs one degree varus.

Similarly, the computer system can detect that patients of a particularsurgeon routinely exhibit positive outcomes (e.g., low pain levels,restoration of range of motion, and/or success during physical therapy).The computer system can then extract trends in the particular surgeon'ssurgical plans and deviations to deduce resected contours, surgicalsteps, and tolerance windows that contribute to these positive outcomes.From these trends, the computer system can adapt future surgical plansfor the particular surgeon and surgical plans for other surgeons toinclude these resected contours, surgical steps, and tolerance windowsthat contribute to the positive outcomes.

Thus, the computer system can correlate surgical outcomes withparticular steps, resected contours, deviations, processes, etc. toinform development of improved surgical plans for each particularsurgeon and for groups of surgeons.

However, the computer system can extract trends from historical surgicaldata and surgical outcomes to inform future surgical plans in any othersuitable way.

17.1 Modeling

In one variation, the computer system aggregates surgical and patientoutcome data and implements machine learning or statistical techniquesto derive a relationship between patient outcomes and various featuresof these patients' surgeries.

In one implementation, the system aggregates surgical input dataincluding: surgical plans for a population of patients on which aparticular type of surgery (e.g., total knee replacements) wereperformed, such as defining target resected contours of tissues ofinterest and target surgical implant positions; actual resected contoursof these tissues of interest and actual surgical implant positionsdetected during these surgeries; numbers of resected contouradjustments; surgical implant types, sizes, and geometries; etc. In thisimplementation, the computer system can also aggregate surgeonidentifiers and patient demographics (e.g., age, gender, weight,pre-operative mobility, pre-operative fitness level, medical history)for surgeons and patients present in each of the surgeries. Furthermore,the computer system can aggregate patient outcome data for each of thesesurgeries, such as: range of motion regained by the patient, such as afunction of time or at a target time (e.g., six months) post-surgery;whether the patient achieved a fully recover (e.g., within six monthspost-surgery); whether the patient experienced a post-operativeprobability of infection; whether a second corrective surgery wasnecessary; patient-reported satisfaction (e.g., from 0% to 100%); etc.for each surgery.

The computer system can then: assemble these surgical inputs, surgeons,and patient data types into one vector (or other data container) perpatient in the population; label each vector with corresponding patientoutcome data; and then implement deep learning, a convolutional neuralnetwork, regression, and/or other machine learning or statisticaltechniques to derive correlations between these surgical inputs andpatient outcomes—corrected or adjusted for surgeon and patientdemographic. The computer system can then store these correlations in apatient outcome model. Later, the computer system can: implement thispatient outcome model to predict an outcome of a next surgery on apatient based on inter-operative surgical input data (e.g., resectedcontours, surgical implant placement) collected during the surgery; andserve feedback or guidance to the surgeon to verify or modify resectionof tissues of interest and/or placement of surgical implants, asdescribed above.

Furthermore, the computer system can: collect surgical input data duringthis surgery; label these surgical input data with patient outcome dataas these patient outcome data become available over time; append acorpus of surgical input data and patient outcome data across thispopulation with these new surgical input data and patient outcome data;and retrain the patient outcome model accordingly.

For example, during a surgery, the computer system can implement methodsand techniques described above to calculate an absolute spatialdifference between the actual resected contour of the hard tissue ofinterest detected in the first sequence of optical scans and theunresected contour of the hard tissue of interest and then record thesedata in association with the patient and surgeon. Later, the computersystem can: label this absolute spatial difference with a post-operativeoutcome of the patient (e.g., the patient's satisfaction, the patient'srecover time); and store the absolute spatial difference in a databasewith a corpus of absolute spatial differences labeled with patientoutcomes for a set of instances of the surgical operation within apopulation of patients. Finally, the computer system can: derive acorrelation between outcomes and absolute spatial differences betweenactual resected contours of the hard tissue of interest and unresectedcontours of the hard tissue of interest within this population ofpatients; and store this correlation in a patient outcome model.

Similarly, the computer system can derive correlations between patientoutcomes and: differences between actual and target resected contours;and/or differences between actual and target surgical implant positions.For example, the computer system can implement methods and techniquesdescribed above to detect and track a spatial difference between targetand actual resected contours for a hard tissue of interest during asurgery. The computer system can then: store this spatial difference ina database with a corpus of spatial differences labeled with patientoutcomes for a set of instances of this surgical operation across apopulation of patients; and then derive a correlation between successfulrecoveries of patients within this population, such as for all patientsor specifically patients operated on by this same surgeon); and spatialdifferences between actual resected contours of the hard tissue ofinterest and target resected contours of the hard tissue of interest,such as specified in surgical plans defined by this same surgeon.

Furthermore, in this variation, the computer system can leverage thispatient outcome model to assist a surgeon in defining a pre-operativesurgical plan for a next patient. For example, as the surgeon develops asurgical plan for the next patient within a physician portal, thecomputer system can: inject target resected contour and surgical implantvalues specified by the surgeon into the patient outcome model topredict effects of these values on the patient's predicted outcome; andthen serve a recommendation to the surgeon for adjustment of thepre-operative surgical plan accordingly.

The computer systems and methods described herein can be embodied and/orimplemented at least in part as a machine configured to receive acomputer-readable medium storing computer-readable instructions. Theinstructions can be executed by computer-executable componentsintegrated with the application, applet, host, server, network, website,communication service, communication interface,hardware/firmware/software elements of a user computer or mobile device,wristband, smartphone, or any combination thereof. Other systems andmethods of the embodiment can be embodied and/or implemented at least inpart as a machine configured to receive a computer-readable mediumstoring computer-readable instructions. The instructions can be executedby computer-executable components integrated by computer-executablecomponents integrated with apparatuses and networks of the typedescribed above. The computer-readable medium can be stored on anycomputer readable media such as RAMs, ROMs, flash memory, EEPROMs,optical devices (CD or DVD), hard drives, floppy drives, or any device.The computer-executable component can be a processor but any dedicatedhardware device can (alternatively or additionally) execute theinstructions.

As a person skilled in the art will recognize from the previous detaileddescription and from the figures and claims, modifications and changescan be made to the embodiments of the invention without departing fromthe scope of this invention as defined in the following claims.

I claim:
 1. A method for tracking and adapting to deviations fromsurgical plans comprising: accessing a virtual patient model defining atarget resected contour of a hard tissue of interest; during a firstperiod of time succeeding resection of the hard tissue of interestwithin a surgical operation: accessing a first sequence of optical scansrecorded by an optical sensor facing a surgical field occupied by apatient; detecting a set of features representing the patient in thefirst sequence of optical scans; registering the virtual patient modelto the hard tissue of interest in the surgical field based on the set offeatures; and detecting an actual resected contour of the hard tissue ofinterest in the first sequence of optical scans; calculating a spatialdifference between the actual resected contour of the hard tissue ofinterest detected in the first sequence of optical scans and the targetresected contour of the hard tissue of interest represented in thevirtual patient model registered to the hard tissue of interest in thesurgical field; and presenting the spatial difference to a surgeonduring the surgical operation.
 2. The method of claim 1: furthercomprising, during an initial period of time preceding the first periodof time and succeeding incision of the patient proximal the hard tissueof interest: accessing an initial sequence of optical scans recorded bythe optical sensor; detecting an unresected contour of the hard tissueof interest in the initial sequence of optical scans; and registeringvirtual hard tissue features defined in the virtual patient model to theunresected contour of the hard tissue of interest; and detecting the setof intermediate features, on the patient and proximal the hard tissue ofinterest, in the initial sequence of optical scans; further comprisingderiving a spatial relationship between the set of features and thevirtual patient model based on registration of the virtual patient modelto the hard tissue of interest; and wherein registering the virtualpatient model to the hard tissue of interest in the surgical field basedon the set of features during the first period of time comprisesregistering the virtual patient model to the hard tissue of interestbased on the spatial relationship and the set of features detected inthe first sequence of optical scans.
 3. The method of claim 2, furthercomprising: calculating an absolute spatial difference between theactual resected contour of the hard tissue of interest detected in thefirst sequence of optical scans and the unresected contour of the hardtissue of interest; labeling the absolute spatial difference with apost-operative outcome of the patient; storing the absolute spatialdifference in a database with a corpus of absolute spatial differenceslabeled with patient outcomes for a set of instances of the surgicaloperation within a population of patients; and deriving a correlationbetween outcomes and absolute spatial differences between actualresected contours of the hard tissue of interest and unresected contoursof the hard tissue of interest within the population of patients.
 4. Themethod of claim 3: wherein deriving the correlation between outcomes andspatial differences comprises deriving the correlation between:successful recoveries within the population of patients; and absolutespatial differences between actual resected contours of the hard tissueof interest and unresected contours of the hard tissue of interestwithin the population of patients; and further comprising, for a secondinstance of the surgical operation planned for a second patient,generating a recommended absolute spatial difference between actualresected contours of the hard tissue of interest and unresected contoursof the hard tissue of interest in the second patient, during the secondinstance of the surgical operation, based on the correlation.
 5. Themethod of claim 2: wherein accessing the virtual patient model comprisesaccessing the virtual patient model defining a virtual representation ofa generic unresected contour of the hard tissue of interest and avirtual representation of a generic target resected contour of the hardtissue of interest; wherein registering virtual hard tissue featuresdefined in the virtual patient model to the unresected contour of thehard tissue of interest comprises: calculating a best-fit location thatminimizes error between the virtual representation of the genericunresected contour of the hard tissue of interest and the unresectedcontour of the hard tissue of interest detected in the initial sequenceof optical scans; morphing the virtual representation of the genericunresected contour of the hard tissue of interest in the virtual patientmodel into conformity with the unresected contour of the hard tissue ofinterest detected in the initial sequence of optical scans; and morphingthe virtual representation of the generic resected contour of the hardtissue of interest in the virtual patient model into conformity with thevirtual representation of the generic unresected contour of the hardtissue of interest in the virtual patient model.
 6. The method of claim2, wherein accessing the virtual patient model comprises: accessing thevirtual patient model comprising a virtual representation of theunresected contour of the hard tissue of interest; during the initialperiod of time, receiving a command from the surgeon specifying a set oftarget resection parameters for the hard tissue of interest; andprojecting the set of target resection parameters onto the virtualrepresentation of the unresected contour of the hard tissue of interestto define the target resected contour of the hard tissue of interest;and storing the target resected contour of the hard tissue of interestin the virtual patient model.
 7. The method of claim 1: furthercomprising, prior to the surgical operation: accessing a pre-operativescan of the hard tissue of interest of the patient; extracting a virtualrepresentation of the unresected contour of the hard tissue of interestfrom the pre-operative scan; generating a virtual representation of thetarget resected contour of the hard tissue of interest based on thevirtual unresected contour of the hard tissue of interest and apre-operative surgical plan defined by the surgeon; compiling thevirtual representation of the unresected contour of the hard tissue ofinterest and the virtual representation of the target resected contourof the hard tissue of interest into the virtual patient model; andstoring the virtual patient model, in association with the patient, in adatabase; and wherein accessing the virtual patient model comprises,during the surgical operation, accessing the virtual patient model fromthe database.
 8. The method of claim 1, further comprising: during thefirst period of time: prompting the surgeon to confirm intent of thespatial difference; and in response to confirmation of intent of thespatial difference from the surgeon, prompting the surgeon to provide areason for the spatial difference; and in response to receipt of thereason from the surgeon, recording the spatial difference, confirmationof intent of the spatial difference, and the reason in a database and inassociation with the surgical operation.
 9. The method of claim 1,further comprising: detecting insufficient material removal from thehard tissue of interest based on the spatial difference; in response todetecting insufficient material removal from the hard tissue ofinterest, serving a prompt to the surgeon to remove additional materialfrom the hard tissue of interest; during a second period of timesucceeding a second resection of the hard tissue of interest within thesurgical operation: accessing a second sequence of optical scansrecorded by the optical sensor; detecting the set of featuresrepresenting the patient in the second sequence of optical scans;registering the virtual patient model to the hard tissue of interest inthe surgical field based on the set of features; and detecting a secondactual resected contour of the hard tissue of interest in the secondsequence of optical scans; calculating a second spatial differencebetween the second actual resected contour of the hard tissue ofinterest detected in the second sequence of optical scans and the targetresected contour of the hard tissue of interest represented in thevirtual patient model registered to the hard tissue of interest in thesurgical field; and presenting the second spatial difference to thesurgeon.
 10. The method of claim 1: wherein accessing the virtualpatient model comprises accessing the virtual patient model defining thetarget resected contour of the hard tissue of interest and defining asecond target resected contour of a second hard tissue of interestadjacent the hard tissue of interest; further comprising: detectingexcessive material removal from the hard tissue of interest based on thespatial difference; in response to detecting excessive material removalfrom the hard tissue of interest, modifying the second target resectedcontour of a second hard tissue of interest to compensate for excessivematerial removal from the hard tissue of interest.
 11. The method ofclaim 10: wherein accessing the virtual patient model comprisesaccessing the virtual patient model defining the target resected contourof the hard tissue of interest comprising a femur and defining a secondtarget resected contour of a second hard tissue of interest comprising atibia; wherein calculating the spatial difference comprises detecting aspatial difference between actual resection of a femoral condyle of thepatient and target resection of the femoral condyle of the patient;wherein detecting excessive material removal from the hard tissue ofinterest comprises detecting resection of excessive material from thefemoral condyle of the patient; and wherein modifying the second targetresected contour of the second hard tissue of interest to compensate forexcessive material removal from the hard tissue of interest comprisesreducing magnitude of resection of a tibial plateau of the patient tocompensate for excessive material removal from the femoral condyle ofthe patient.
 12. The method of claim 1, further comprising: labeling thespatial difference with a post-operative outcome of the patient; storingthe spatial difference in a database with a corpus of spatialdifferences labeled with patient outcomes for a set of instances of thesurgical operation within a population of patients; and deriving acorrelation between outcomes and spatial differences between actualresected contours of the hard tissue of interest and target resectedcontours of the hard tissue of interest within the population ofpatients.
 13. The method of claim 12: wherein storing the spatialdifference in the database with the corpus of spatial differenceslabeled with patient outcomes for the set of instances of the surgicaloperation within the population of patients comprises aggregating thecorpus of spatial differences labeled with patient outcomes for the setof instances of the surgical operation performed by the surgeon; whereinderiving the correlation between outcomes and spatial differencescomprises deriving the correlation between: successful recoveries withinthe population of patients operated on by the surgeon; and spatialdifferences between actual resected contours of the hard tissue ofinterest and target resected contours of the hard tissue of interestspecified in surgical plans defined by the surgeon; and furthercomprising, for a second surgical plan defined by the surgeon for asecond instance of the surgical operation planned for a second patient,serving a recommendation for adjustment of the second surgical plan,based on the correlation, to the surgeon prior to the second instance ofthe surgical operation.
 14. The method of claim 1, further comprising,during the first period of time: selecting a frame in the first sequenceof optical scans; projecting a virtual representation of an unresectedcontour of the hard tissue of interest, defined in the virtual patientmodel, onto the frame; writing the spatial difference to the frame; andserving the frame to a physician portal affiliated with a second surgeonlocated remotely from the surgical field.
 15. The method of claim 1:wherein accessing the virtual patient model comprises accessing thevirtual patient model defining the target resected contour of a femoralcondyle of a femur of the patient; wherein detecting the actual resectedcontour of the hard tissue of interest in the first sequence of opticalscans comprises detecting the actual resected contour the femoralcondyle of the patient in the first sequence of optical scans; andwherein detecting the spatial difference comprises detecting the spatialdifference between the actual resected contour of the femoral condyledetected in the first sequence of optical scans and the target resectedcontour of the femoral condyle defined in the virtual patient model. 16.The method of claim 15: wherein calculating the spatial differencecomprises: calculating a distance magnitude difference between theactual resected contour of the femoral condyle and the target resectedcontour of the femoral condyle; calculating an orientation differencebetween the actual resected contour of the femoral condyle and thetarget resected contour of the femoral condyle; and characterizing asurface profile difference between the actual resected contour of thefemoral condyle and the target resected contour of the femoral condyle;and wherein presenting the spatial difference to the surgeon during thesurgical operation comprises rendering the distance magnitudedifference, the orientation difference, and the surface profiledifference on a display present near the surgical field.
 17. The methodof claim 16, wherein rendering the distance magnitude difference, theorientation difference, and the surface profile difference on thedisplay comprises, during the second period of time: detecting aposition of an augmented reality headset, worn by a surgeon andcomprising the display, proximal the surgical field; estimating aperspective of the surgeon viewing the surgical field based on theposition of the augmented reality headset; generating an augmentedreality frame comprising a projection of the target resected contour ofthe hard tissue of interest of the patient, defined in the virtualpatient model, from the perspective of the surgeon; inserting thedistance magnitude difference, the orientation difference, and thesurface profile difference into the augmented reality frame; and at theaugmented reality headset, rendering the augmented reality frame. 18.The method of claim 1, further comprising: calculating an absolutespatial difference between the actual resected contour of the hardtissue of interest detected in the first sequence of optical scans andan unresected contour of the hard tissue of interest defined in thevirtual patient model; accessing a correlation between outcomes andabsolute spatial differences between actual resected contours of thehard tissue of interest and unresected contours of the hard tissue ofinterest within a population of patients subject to instances of thesurgical operation; predicting a probability of successful outcome ofthe patient based on the absolute spatial difference and thecorrelation; in response to the probability of successful outcomeexceeding a threshold probability, prompting the surgeon to move to anext step of the surgical operation; and in response to the probabilityof successful outcome falling below the threshold probability, promptingthe surgeon to correct the actual resected contour of the hard tissue ofinterest of the patient to reduce the spatial difference.
 19. A methodfor tracking and adapting to deviations from surgical plans comprising:accessing a virtual patient model defining a target position of anartificial implant on a tissue of interest; during a first period oftime succeeding placement of the artificial implant on the tissue ofinterest within a surgical operation: accessing a first sequence ofoptical scans recorded by an optical sensor facing a surgical fieldoccupied by a patient; detecting a set of features representing thepatient in the first sequence of optical scans; registering the virtualpatient model to the tissue of interest in the surgical field based onthe set of features; and detecting an actual position of the artificialimplant on the tissue of interest in the first sequence of opticalscans; calculating a spatial difference between the actual position ofthe artificial implant on the tissue of interest detected in the firstsequence of optical scans and the target position of the artificialimplant on the tissue of interest represented in the virtual patientmodel registered to the tissue of interest in the surgical field; andpresenting the spatial difference to a surgeon during the surgicaloperation.
 20. The method of claim 19, further comprising: calculatingan absolute position of the artificial implant on the tissue of interestdetected in the first sequence of optical scans; accessing a correlationbetween outcomes and absolute positions of instances of the artificialimplant on the tissue of interest within a population of patientssubject to instances of the surgical operation; predicting a probabilityof successful outcome of the patient based on the absolute position ofthe artificial implant on the tissue of interest and the correlation; inresponse to the probability of successful outcome exceeding a thresholdprobability, prompting the surgeon to move to a next step of thesurgical operation; and in response to the probability of successfuloutcome falling below the threshold probability, prompting the surgeonto adjust the absolute position of the artificial implant on the tissueof interest.